Application in Reinforcement Learning for Diagnosis Based on Medical Image

Reinforcement learning (Sutton & Barto, 1998) is a formal mathematical framework in which an agent manipulates its environment through a series of actions and in response to each action receives a reward value. The agent will try to maximize results by choosing the best reward with the internal agent. In the fact, the agent aims to get a maximum reward over time. The agent is not taught to decide which road chooses but some signals are given to him to allow decide the best road.

Reinforcement Learning: Supervised or Unsupervised?

As we know supervised learning essence is that you teach your model you give it a labeled input and you control your output, and the other hand unsupervised learning is mean that you let your model detect the abnormal actions, data, and objects, in your system by using some algorithms which your model to label data itself. In RL there is no teacher or algorithm because the agent relies on itself, by doing action and then seeing the effect positive or negative.

How does RL work?

The Principe of RL is inspired by children’s learning, when we’re in a complex situation where the environmental knowledge is limited, and the agent’s behavior is unknown too, children execute random actions in such space to discover the world around them, they move randomly, take things, manipulate space (food, water, electric, people reaction…) for example is he did something and her parent being not happy he understands that he did a bad thing.

The agent interacts with his environment step by step, in each step t = 0, 1, 2, 3, … the agent precepts that the environment is in st state and it makes the action at, the environment makes a transition to st+1 state and emits a reward of rt+1, the agent looks to maximize the reward.

We can formulate this task with this function: RL(s, a) = rt ɛ R, the agent receives this result of the t step and try to maximize it for the next step t+1

Note that the RL goal must be indicated from the beginning, for example when playing draughts the player looks to win the game and not lose, so this is the goal of RL and the problem is to choose the policy of the action which maximizes the totality of the rewards received by the agent. An actions policy corresponds to the function: π(s) = a means that the state generates the action which is executed by the agent. This later searches the policy which maximizes the sum of rewards is called π*

Q-Learning:

Q learning is the formulation of Reinforcement learning, the Capital letter Q corresponds to the quality of learning as well as the reinforcement function, for each state the agent looks to maximize his reward total, so to do this the latter must initiate reward sum and start to do action and estimate reward, Then, with each choice of action, the agent observes the reward and the new state (which depends on the previous state and the current action). The heart of the algorithm is an update of the value function. The definition of the value function is updated at each step as follows:

Where s’ is the new state and s is the previous state a is the action chosen by the agent and r is the reward received, α is the learning factor, γ is the discount factor.

How α, γ work?

As we said α is the learning factor, so it represents the learning level of our agent, if α = 0 that means that the new state is the same previous, then the agent will not consider it, else if α =1 that means that the new state has different with the previous and then the agent will consider it

And the other hand γ is the discount factor that considers the importance of the next reward if the γ is near 0 this makes the agent myopic, then it will consider the current rewards and not the next. A γ near 1 may diverge the Q sum.

Medical imaging has made a revolution in the medical field, making it possible to execute diagnosis without any invasion with perfect accuracy and very fast time. Medical images lead us to broaden our observation capabilities and allow us to understand the biophysical world, by applying new algorithms such as new protocols for manipulation which help to increase accuracy with the intervention of artificial intelligence.

Since the discovery of the X-Ray in 1885, medical imaging is a fast way to acquire information about a patient’s health state. During the last two decades, Significates advances are noticed in computerized medical imaging, and two and three dimensions modalities which became more helpful for radiologists.

While in the last century, Radiology imaging was the first and unique way to the acquisition of images, new modalities are developed including physiological, and metabolic state detection, the most know techniques are Computed Tomography (CT), Magnetic Resonance Imagery (MRI), Positron Emission Tomography (TEP) and Ultrasound.

Today imaging medical become an essential task in the medical routine, in most cases anomalies can be observed before appearing of symptoms and this is a very important point. Each imaging modality has a particular organ a tissue to observe, but they are complete and offer and full view of such organs from the functional and structural sides.

We can divide all modalities into 2 categories: structural or morphological and functional

For the first, we found: X-Ray, Ultrasound, MRI, Computed tomography CT

The second time, we have multiple technologies which are used to assess tissue metabolism, like as Scintigraphy, single photon emission computerized tomography (SPECT), positron emission tomography (PET), and functional magnetic resonance imagery (fMRI). in the next section, we will talk about each technology.

The X-Ray is produced in 2 ways: the first one is that a high difference in electricity in the X-ray tube between the anode and cathode, this late is associated with the filament which issued electrons also will react in two ways, first by Bremsstrahlung where the electron is decreased his speed and converted it to X-Ray energy, second is Characteristic radiation where the electron emitted by filament remove an electron of nuclei which create gap this will be occupied by the supra electron which gives emitted radiation outside the nuclei, this is X-RAY

X-Ray is used to be projected onto the patient’s body, and they transverse the targeted area, bones can block these rays that we observe as light areas, on another side soft tissue allows passage of these Rays which appear as a dark area in the image.

Ultrasound imaging is the second most popular imaging modality after X-Ray, is estimated that 25% of the medical process call Ultrasound Imaging(1), and this modality complements other modalities such as MRI and CT. Ultrasound is based on high-frequency sound waves and for that, they are called Ultrasound, these late waves are used to traverse tissues and produce different echoes and categorize each tissue type, however, the echoes will be captured by a receiver and sent to the computer which transforms it to screen signal.

Second, Computed Tomography or CT is an imaging technology that uses X(Rays for the reconstruction of a 3D image of a patient’s body by capturing images from different angles, first the image is generated by several 2D radiographs, then we use an algorithm called Random Transfer. CT offers high contrast between soft tissue and bone and low contrast among soft tissue.

Now, we’ll talk about MRI technology which refers to Magnetic Resonance imaging, in this technique we use one of the most popular and important proprieties of the human body which is composed of about 75% water, so when it is placed within a magnetic field the hydrogen nuclei will be excited, however, this will make them in the supra-energetic state which needs to spend this extra-energy under relaxation, this late depends on such tissue type also we can create images by using this difference. An advantage of MRI is that it is not inosinate so there is no hazard for the patient. MRI is so better for soft tissue in particular brain and spinal cord imaging.

In the next post, we’ll talk a bit more about functional imaging modalities.

Written by Said Jadidi – Originally published here.