Class-adapted blind image deblurring (by Marina)

In the previous post, I introduced the concept of blind image deblurring: the problem when we have an observed blurred image and we want to find an underlying sharp image and a blurring filter.  Also, I mentioned that there are different types of blurring filters and they have different characteristics. What I did not mention is that there are also different types of sharp images (figure 1).

Figure 1: Different types of images.

Different types of sharp images can have completely different structure. For example, look at a (document like) text image which has sharp transitions and usually comes in two tones, black and white. On the other side, a natural image has smooth regions (like sky or lake), texture (bridge, trees) and some sharp edges (boat). An image of a face can have small texture (like area below the eyes), that is very difficult to estimate ones when is blurred.

As we have a problem where we need to estimate two unknowns (an underlying sharp image and a blurring filter) from only one observation (a blurred image), we usually introduce some kind of prior knowledge on both the sharp image and the blurring filter. A prior knowledge of image can represent a characteristic or a set of characteristics about sharp images itself. The similar is for blurring filters. But, as you know so far, there are different types of images and different types of blurs, so how to find a unique prior knowledge (one for an image and one for a blur) that can cover these differences?

We proposed one solution in the paper presented at the International Conference on Image Processing (ICIP) 2017. The main idea behind the proposal is that we can learn an image prior for different types of images. For example, if we know that we have a blurred image of text, we can use sharp images that contain text (other images) to learn some underlying characteristics of text images.

Figure 2: Clean image of text, blurred image of text and our estimated image.

More information can be found here: https://arxiv.org/abs/1709.01710

Image deblurring (by Marina)

First, I want to apologize for our (little bit longer) disappearance from the blog. I hope that with next several posts you will see the reason behind it.
Here, I want to start with an introduction of my specific topic, blind image deblurring. After the introduction, following posts will be about some of my recent publications, a little bit more technical, but still, I hope, interesting.

What is image deblurring?
It is a process where you have an observed blurred image and you want to estimate, to find, a sharp original image hidden “behind” the blurred image (figure 1).

Figure 1: Image deblurring.

The blurring of the image can have different reasons, like camera shake or out-of-focus, and it is usually described with the point spread function (PSF) or a blurring operator (filter). As there are different reasons for blurry image, there are different blurring filters (figure 2).

Figure 2: Blurring filters: Gaussian, linear motion, out-of-focus, uniform and nonlinear motion blur.

There are two types of image deblurring:
1.    Non-blind image deblurring – where we know how the blurring filter looks like.
2.    Blind image deblurring – when we have a partial knowledge or no knowledge at all about the blurring filter.

The second problem is more realistic one (because usually, we do not know how the blurring occurs) and it is the one that I am interested in. Also, blind image deblurring is more complicated to solve, because in this case we only have a blurred image (one that we observed), and we need to find an underlying sharp image and a blurring filter at the same time. Also, an additional problem that we have is a presence of noise, like we explained in one of the previous posts, that makes our job even harder.

Phase-contrast Imaging

Phase-contrast imaging is one of the highest resolution imaging technique which is widely used in many different areas. This imaging technique exploits the fact that different material has a different refractive index. The light (EM wave) passing through a different material under observation, will be bent according to the respective refractive index of the material. This introduces phase variation which is detected to infer the required physical parameters of the material.

Phase-contrast imaging has got a wide application in biological, medical and geological science. Let us glance through two important applications of it, namely in X-Ray Imaging and Transmission Electron Microscopy (TEM).

X-Ray Imaging

X-rays imaging is performed by measuring differences in the attenuation of X-rays by the tissues. The material properties are inferred from the differences in the number of photons detected at the end of defined paths in the sample. Naturally, then this form of X-ray imaging concentrates almost entirely on the particle side of X-ray behaviour. This can be considered as a photo-electric effect. The main drawback of photo-electric effect based X-ray imaging is that the image contrast depends on the atomic number and density of the tissue. For instance, let us consider a bone and soft tissue. Here the conventional X-ray provides good contrast as there is a considerable difference in the atomic number and density of bone and soft tissue. But what happens if we consider thin tissues with a very similar atomic number and only small density variation? Obviously the resulting image- contrast will be poor.

X-ray phase-contrast image of spider [1]

Phase-contrast imaging tries to address this issue. It makes use of the fact that X-rays also exhibit wave-like behaviour in addition to the particle nature.  When the X-ray waves interact with a material, the interactions induce a phase shift in the wave. We have sophisticated technology and well-developed detectors available today to detect these phase shifts. The phase-contrast based X-ray imaging shows much better performance in terms of image contrast compared to the conventional X-ray imaging.  

Transmission electron microscopy (TEM)

Transmission electron microscopy (TEM) is a microscopy technique in which a beam of electrons is transmitted through an ultra-thin specimen, interacting with the specimen as it passes through it. These interactions are captured in the form of an image onto an imaging device.

In addition to the contrast in the transmitted beam due to the electronic interaction, we can introduce diffraction contrast by applying the principle of phase-contrast imaging. This ability arises from the fact that the atoms in a material diffract electrons as the electrons pass through them, causing diffraction contrast.

  A TEM image of the polio virus [2]

References


[1]  https://en.wikipedia.org/wiki/Phase-contrast_imaging

[2]  https://en.wikipedia.org/wiki/Transmission_electron_microscopy

[3]  http://users.ox.ac.uk/~atdgroup/technicalnotes/Phase%20Constrast%20X-ray%20imaging.pdf

Medical image restoration

One of the arguably most important applications where we use image restoration is medical imaging. We are talking about almost all kinds of medical imaging (radiography,  magnetic resonance imaging (MRI), tomography and so on). 

Why we said it is “arguably most important”? Most important because everybody (doctors, engineers, patients) can recognize the very importance of the good medical image in diagnostic (and control of disease progression) process. Arguably, because there are other very important applications where we use image restoration so it is hard to make a fair comparison. Our goal here is not to try to do that. 

Here, like in many of our previous topics, we will try to introduce the problem itself through several examples. 

• Some of the artifacts in medical images arise because of the very device that we are using for taking images. Probably, the best representative of this problem is the presence of noise. Beginners in image processing soon discover that a denoising step is often required before any relevant information can be extracted from an image. That is because noise can “cover” some important structure in the image or for example destroy existing edges in the image, important for a proper diagnostic process. 

Noisy and denoised image of the brain [1] 

• The other example can be existing of the blur in medical images. Blur occurs because of the device we are using (like out-of-focus blur) but also because of some other effects that sometimes we cannot control such as movements of a patient. For many medical images, it is not recommended to repeat a process of taking the image because of safety (radiography radiation), time (long duration of a procedure itself) or even money, so doctors need to use what they have. And sometimes the only thing they have is the image with the blurry artifacts. 

Deblurring real-degraded Brain CT images: (C1) naturally blurred full-size CT images; (C2) zoomed-in portions; (C3) images from top to bottom are deblurred by Zohair filter [2]

References:

[1] P. Coupé, P. Yger, S. Prima, P. Hellier, C. Kervrann, C. Barillot, An Optimised Blockwise NonLocal Means Denoising Filter for 3-D Magnetic Resonance Images, IEEE Transactions on Medical Imaging, 27(4):425–441, 2008.

[2] Z. Al-Ameen, G. Sulong, A novel Zohair filter for deblurring computed tomography medical images. International Journal of Imaging Systems and Technology 25(3), 2015.

Image restoration techniques in surveillance applications

So far we mentioned several important applications where image restoration techniques are valuable. One of them is definitely surveillance, but let’s clear up something at the beginning. When we are talking about surveillance we are not thinking on spying. Indeed spying is part of a surveillance, but only a small part. Surveillance is important as an infallible part of the protection of the people and properties. It is part of the traffic control and safety, law enforcement, forensic, environmental protection (like wildlife monitoring) and of course military. Here we are only focused on video (or image) based surveillance.

Let start with mentioning several very important problems that restoration techniques are trying to solve:

• Video/Image blurred due to motion of the vehicle or/and people (and camera)

It is clear that, for example, in video surveillance of vehicles the very subject of interest (vehicle) is moving and additionally camera can move due to the influence of wind or vibration. All these movements can cause the blur effect in video, so-called, motion blur. In many cases, because of the motion blur, we are not able to detect the subject of interest correctly, for example, we are not able to read numbers and letters on the car plate. Here we can use deblurring techniques to restore this video and for example, read car plates.

Blurred and deblurred car plates [1]

• Video/Image noisy and/or blurred due to equipment used

Here we are talking about videos (and images) noisy and/or blurred due to equipment used for filming. We all know that a good equipment is expensive and that a cheaper one sometimes is not giving the desired result. Here again, we can combine the equipment that we have with restoration (denoising and deblurring) techniques.

Corrupted and denoised video [2]

• Extreme zoom surveillance (astronomy and wildlife monitoring)

Applications of digital imaging with extreme zoom are traditionally found in astronomy and wildlife monitoring. For example, you want to film a wild animal during some sensitive moment, like feeding, but you can not come to close. Usually, in that situation, people use extreme zooming. You can imagine that video or image that you take (especially if you do not use a very good camera) will be nearly useless and again, here, image restoration methods can help.

Extreme zoomed image of face [3]

• Video/Image artefacts as result of the compression 

Here, we have a problem with lack of storage space. It is not hard to imagine that if we are taking hours and hours of videos, in one moment we will spend all our storage space. In some application, you can delete videos that are old several days, but what we can do if all our collected videos are precious and we can not delete them? In that case, you can use compression techniques to compress your videos so they take a less space. During compression, you will surely lose some information that maybe can be retrieved by using some of the video restoration methods.  

References:

[1] Svoboda, P., Hradis, M., Marsik, L., & Zemcik, P. (2016). CNN for license plate motion deblurring. arXiv preprint arXiv:1602.07873.

[2] V. Cheung, B. J. Frey, and N. Jojic.  Video epitomes.  Intern. Journal of Computer Vision (IJCV), 2007.

[3] Y. Yao, B.R. Abidi, M.A. Abidi. Extreme Zoom Surveillance: System Design and Image Restoration. Journal of Multimedia, 2007.