Application in Reinforcement Learning for Diagnosis Based on Medical Image

Reinforcement learning (Sutton & Barto, 1998) is a formal mathematical framework in which an agent manipulates its environment through a series of actions and in response to each action receives a reward value. The agent will try to maximize result by choosing the best reward with internal agent. In the fact the aim of the agent is to get a maximum reward over time. The agent is not taught to decide which road chooses but some signals are given to him to allow decide the best road.

Reinforcement Learning: Supervised or Unsupervised?

As we know supervised learning essence is that you teach your model you give it a labeled input and you control your output, and the other hand the unsupervised learning is mean that you let your model detect the abnormal actions, data, object, in your system by using some algorithms which your model to label data itself. In RL there is no a teacher or algorithm because the agent is relies on itself, by doing action and then see the effect positive of negative.

How RL work?

The Principe of RL is inspired from children’s learning, when we’re before a complex situation where the environment knowledge is limited, and the agent’s behavior is unknown too, children executes random actions in such space to discover the world around, he move randomly, take things, manipulate space (food, water, electric, people reaction…) for example is he did something and her parent being not happy he understand that he did a bad things.

The agent interact with his environment step by step, in each step t = 0, 1, 2, 3, … the agent precepts that the environment is in st state and it make the action at, the environment make an transition to st+1 state and emits an reward of rt+1, the agent look to maximize the reward.

We can formulate this task with this function: RL(s, a) = rt ɛ R, the agent receive this result of the t step and try to maximize it for the next step t+1

Note that the RL goal must be indicated from the beginning, for example when playing draughts the player look to win the game and not loss, so this is the goal of RL and the problem is to choose the actions policy which maximize the totality of the rewards received by the agent. An actions policy corresponds to the function: π(s) = a means that the state generates the action which is executed by the agent. This later search the policy which maximizes the sum of rewards its called π*


The Q learning is the formulation of an Reinforcement learning, the Capital letter Q correspond at the quality of learning as well as the reinforcement function, for each state the agent look to maximize his reward total, so to do this our this later must initiate reward sum and start to do action and estimate reward, Then, with each choice of action, the agent observes the reward and the new state (which depends on the previous state and the current action). The heart of the algorithm is an update of the value function. The definition of the value function is updated at each step as follows:

Image for post

Where s’ is the new state and s is the previous state a is the action chosen by the agentand is the reward received, α is learning factor, γ is the discount factor.

How α , γ work ?

As we said α is learning factor, so it represents learning level of our agent, if α = 0 that’s mean that the new state is the same previous, then the agent will not consider it, else if α =1 that’s mean that the new state has different with the previous and then the agent will consider it

And the other hand γ is the discount factor it considers the importance if the next reward if the γ is near to 0 this makes the agent myopic, then it will consider the currents rewards and not the next. A γ near to 1 may diverge Q sum.

Image for post

Medical imaging has made a revolution for medical filed, making possible to execute diagnosis without any invasion with perfect accuracy and very fast time. Medical image lead us to broaden our observation capabilities, and allow us to understand biophysical world, by applying new algorithms such as new protocols for manipulation which help to increase accuracy with intervention of artificial intelligence.

From discovery of X-Ray in 1885, medical imaging is the fast way to acquiring information from patient’s health state. During last two decades Significates advances are noticed in computerized medical imaging, two and three dimensions modalities which became more helpful for radiologists.

While in last century, Radiology imaging were the first and unique way to acquisition of images, new modalities are developed include physiological, metabolic state detection, the most know techniques are: Computed Tomography (CT), Magnetic Resonance Imagery (MRI), Positron Emission Tomography (TEP) and Ultrasound.

Today imaging medical become an essential task in medical routine, in most cases anomalies can be observed before appear of symptoms and this is a very important point. Each imaging modality has particular organ a tissue to observe, but they are completely and offer and full view for such organ from functional and structural side.

We can divide all modalities into 2 categories: structural or morphological and functional

For the first we found: X-Ray, Ultrasound, MRI, Computed tomography CT

In the second time we have multiple technologies which are used to assess tissues metabolism, like as Scintigraphy, single photon emission computerized tomography (SPECT), positron emission tomography (PET), and functional magnetic resonance imagery (fMRI). in the next section we will talk about each technology.

The X-Ray are the produced by 2 ways : the first one is that a high difference electric in X ray tube between anode an cathode, this late is associated to filament which issued electrons also they will reacted with two ways, first by Bremsstrahlung where the electron is decrease his speed and converted it to energy which is X-Ray, second is Characteristic radiation where the electron emitted by filament remove an electron of nuclei which create gap this will be occupied by the supra electron which give emitted a radiation outside the nuclei, this is X-RAY

Image for post

X-Ray are used to be projected to patient body, they transverse targeted area, bones can block these Ray for that we observe them as light area, in other side soft tissue allow passage of these Rays this appear as dark area in the image.

Ultrasound imaging is the second most popular imaging modality after X-Ray, is estimated that 25% of medical process call Ultrasound Imaging(1), this modality complement other modalities such as MRI and CT. Ultrasound is based on high frequency sound waves and for that they are called Ultrasound, these late waves are used to traverse tissues and produce different echoes categorize each tissue type, however the echoes will be captured by a receiver and send to computer which transform it to screen signal.

Image for post

Second, Computed Tomography or CT is a imaging technology use X(Rays for reconstruction a 3D image of patient body by capturing images in different angles, first the image is generated by a number of 2D radiographs, then we use an algorithm called Random Transfer. CT offers high contrast between soft tissue and bone, and low contrast among the soft tissue.

Image for post

Now, we’ll talking about MRI technology which refer to Magnetic Resonance imaging, in this technique we use one of the most popular and important propriety of human body that its composed of about 75% water, so when its placed within a magnetic field the hydrogen nuclei will be excited, however this will make them in supra-energetic state which need to spend this extra-energy under relaxation, this late is depend on such tissue type also we can create images by using this difference. An advantage of MRI is that it not inosinate so there is no any hazard for patient. MRI is so better for soft tissue in particular brain and spinal cords imaging.

In the next post we’ll talking bit more about functional imaging modalities.

Written by Said Jadidi – Originally published here.

Leave a reply:

Your email address will not be published.