Mobile Imaging

02 Details

Smartphone microscopy and image analysis for diagnostics

Juliane Pfeil1, Marcus Frohme1, Katja Schulze2

1Division Molecular Biotechnology and Functional Genomics, Technical University of Applied Sciences, Wildau, Germany

2Oculyze GmbH, Wildau, Germany

In the last 10 years, there has been a significant improvement of the optical camera components in mobile devices, as well as an extensive improvement towards their computing power. Both improvements were basic prerequisites to use smartphones not only to make phone calls and browsing, but to also consider them as sensor devices for new applications.

Due to its specific feature of mobility, particularly interesting are insufficiently equipped or remote locations and applications that require immediate analysis. The idea to develop a microscope based on the camera module of a smartphone targeted primarily applications in the medical-diagnostics in order to carry out an on-site and immediate analysis. This so-called point-of-care-testing (POCT) gains not only in developing and emerging countries attention, but also in more rural regions in Germany with a deficient medical infrastructure due to a lack of physicians coming along with an increasing average age of the population (Kassenärztliche Bundesvereinigung 2018a, b).

The first smartphone microscope was presented in 2009 (Breslauer et al., 2009). This very bulky attachment, consisting of various conventional microscope modules, made it possible for the first time to examine cell or tissue samples directly on site. Various research groups subsequently worked on improving the technology in order to not only use the optics of the smartphone, but to also establish automated image analysis, which makes specialist knowledge obsolete. These latest developments allow to examine samples not only in a bright field, but also to establish fluorescence, dark field and phase contrast microscopy. Their application for the diagnosis of various diseases such as tuberculosis, parasite infections and fertility examinations were presented. So far, however, only few solutions are on the market with direct applications for POCT not being available yet. (Pfeil et al., 2018)

In image analysis, with the help of neural networks and Deep Learning (DL) large numbers of different objects in a single picture can be identified. Primarily, these algorithms are used for the detection of common objects, to train self-driving vehicles or to recognize faces. However, their establishment in medical settings, especially in microscopy, is on the advance. The particular difficulty in the recognition of microscopic objects lies not only in the classification of individual objects, but also in achieving a so-called instance segmentation of cells or particles in order to provide the obligatory accuracy for medical-diagnostic applications. (Felsberg, 2017)

Figure 1 exemplifies these differences for microscopic images of blood cells. The easiest approach is a simple classification, i. e. the assignment of a single object or condition to a class. Here one can review if the figure on the left shows a blood sample or something else. However, this statement is not very helpful in a diagnostic context. A localization and classification of all cells, as shown in the middle, opens up the possibility to evaluate each object individually. In order to obtain information about the morphology, the instance segmentation shown on the right is necessary, which is algorithmically the most complex to implement.

blood-sample

Figure 1: Stained blood sample at a magnification of 400x, taken with a laboratory microscope (Keyence BZ 9000). Hemacolor® chemicals were used, which stain red blood cells violet and white blood cells dark violet.

At the Technical University of Applied Sciences in Wildau (TUAS Wildau) in the division „Molecular Biotechnology and Functional Genomics“ the PhD project Planktovision for the identification of different phytoplankton species to evaluate freshwater samples using automatic microscopy and image recognition was successfully carried out (Schulze et al., 2013). Herein it became obvious that a mobile, affordable microscope with automated analyses on site, would offer significant advantages.

To achieve this goal, the company Oculyze, a successful spin-off of TUAS Wildau, has developed a mobile smartphone microscope (Figure 2). The optics of the microscope enables a magnification of 400x for brightfield examinations.

Figure 2: The Oculyze microscope showing a blood sample treated with Hemacolor® at a magnification of 400x.

blood-sample-mobile

Captured images can be evaluated directly and automatically without the need for specialist knowledge.

In human diagnostics, the examination of the blood has high significance. A hematologist can quantify white and red blood cells, differentiate cell groups and/or detect morphological changes and draw conclusions about the general state of health and the immune system of a patient, examine deficiencies and diseases such as anemia and leukemia and detect blood parasites. The long-term goal of MoMiSmart (Mobile Microscopy on the Smartphone) – a subproject of digilog in TUAS Wildau works closely together with Oculyze GmbH – is to automatically detect and assess these indicators using a mobile microscope. The determination of the concentration and size of red blood cells, as well as a model to calculate the hematocrit content (Udroiu, 2014) have already been established using various image analysis algorithms. The results can be displayed directly in a smartphone application. In order to simplify handling in POCT, a movable sample holder was designed to evaluate as many cells of a blood sample as possible and to test the preparation of the blood samples using pre-colored slides.

In order to achieve the highest possible diagnostic accuracy, the complete image analysis will be based on DL algorithms in the future. Only the earlier mentioned instance segmentation permits a reliable analysis of cell morphology and type. This demanding task is performed in a cloud environment on an external server in order to enable the fastest possible evaluation. Due to the central availability of the data, their transfer to a specialist is also unproblematic in order to verify the result and/or to offer a therapy. Furthermore, a plan is in place to develop a specific hardware that can be used independent to the type of mobile device, not only to make the sample stage mobile, but to also motorize it and improve the optics for even better resolution and magnification. The functionality of this POCT system is shown in Figure 3.

planned

Figure 3: The planned, mobile microscope is compatible with most android mobile phones and tablets via a USB connection. The microscopic images are transmitted encrypted to a cloud server and evaluated using powerful DL algorithms. The results are transferred directly to the mobile device or to a physician/hospital.

References

1. Kassenärztliche Bundesvereinigung. Ärztemangel. http://www.kbv.de/html/themen_1076.php (24.10.18)
2. Kassenärztliche Bundesvereinigung. Mehr Pflegebedürftige in Regionen mit hohem Durchschnittsalter der Bevölkerung. http://gesundheitsdaten.kbv.de/cms/html/17050.php (24.10.18)
3.
Pfeil J, Dangelat LN, Frohme M, Schulze K. Smartphone based mobile microscopy for diagnostics. The Journal of Cellular Biotechnology (in press)
4.
Breslauer D, Maamari R, Switz N, Lam W, Fletcher D. Mobile phone based clinical microscopy for global health applications. PloS one. 2009 Jul 22; 4(7):e6320.
5.
Felsberg M. Five years after the Deep Learning revolution of computer vision: State of the art methods for online image and video analysis. Linköping University Electronic Press; 2017.
6.
Schulze K, Tillich UM, Dandekar T, Frohme M (2013) PlanktoVision- an automated analysis system for the identification of phytoplankton. BMC Bioinformatics 2013 14:115. http://dx.doi.org/10.1186/1471-2105-14-115
7.
Udroiu I. Estimation of erythrocyte surface area in mammals. arXiv preprint arXiv:1403.7660. 2014 Mar 29.

03 Comments

Leave a Reply

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.

de_DEGerman
en_GBEnglish de_DEGerman