Smartphone microscopy and image analysis for diagnostics

Juliane Pfeil1, Marcus Frohme1, Katja Schulze2
1Division Molecular Biotechnology and Functional Genomics, Technical University of Applied Sciences, Wildau, Germany 2Oculyze GmbH, Wildau, Germany
2Oculyze GmbH, Wildau, Deutschland

In the last 10 years, there has been a significant improvement of the optical camera components in mobile devices, as well as an extensive improvement towards their computing power. Both improvements were basic prerequisites to use smartphones not only to make phone calls and browsing, but to also consider them as sensor devices for new applications. Due to its specific feature of mobility, particularly interesting are insufficiently equipped or remote locations and applications that require immediate analysis. The idea to develop a microscope based on the camera module of a smartphone targeted primarily applications in the medical-diagnostics in order to carry out an on-site and immediate analysis. This so-called point-of-care-testing (POCT) gains not only in developing and emerging countries attention, but also in more rural regions in Germany with a deficient medical infrastructure due to a lack of physicians coming along with an increasing average age of the population (Kassenärztliche Bundesvereinigung 2018a, b). The first smartphone microscope was presented in 2009 (Breslauer et al., 2009). This very bulky attachment, consisting of various conventional microscope modules, made it possible for the first time to examine cell or tissue samples directly on site. Various research groups subsequently worked on improving the technology in order to not only use the optics of the smartphone, but to also establish automated image analysis, which makes specialist knowledge obsolete. These latest developments allow to examine samples not only in a bright field, but also to establish fluorescence, dark field and phase contrast microscopy. Their application for the diagnosis of various diseases such as tuberculosis, parasite infections and fertility examinations were presented. So far, however, only few solutions are on the market with direct applications for POCT not being available yet. (Pfeil et al., 2018)

In image analysis, with the help of neural networks and Deep Learning (DL) large numbers of different objects in a single picture can be identified. Primarily, these algorithms are used for the detection of common objects, to train self-driving vehicles or to recognize faces. However, their establishment in medical settings, especially in microscopy, is on the advance. The particular difficulty in the recognition of microscopic objects lies not only in the classification of individual objects, but also in achieving a so-called instance segmentation of cells or particles in order to provide the obligatory accuracy for medical-diagnostic applications. (Felsberg, 2017) Figure 1 exemplifies these differences for microscopic images of blood cells. The easiest approach is a simple classification, i. e. the assignment of a single object or condition to a class. Here one can review if the figure on the left shows a blood sample or something else. However, this statement is not very helpful in a diagnostic context. A localization and classification of all cells, as shown in the middle, opens up the possibility to evaluate each object individually. In order to obtain information about the morphology, the instance segmentation shown on the right is necessary, which is algorithmically the most complex to implement.
In Abbildung 1 sind diese Unterschiede anhand einer mikroskopischen Aufnahme von Blutzellen dargestellt. Der einfachste Ansatz ist eine simple Klassifikation, d. H. die Zuordnung eines einzigen Objekts oder Zustands zu einer Klasse. In diesem Beispiel könnte überprüft werden, ob die Abbildung auf der linken Seite eine Blutprobe zeigt oder etwas anderes. Diese Aussage ist im diagnostischen Kontext jedoch wenig hilfreich. Eine Lokalisation und Klassifikation aller Zellen, wie im mittleren Bild, eröffnet die Möglichkeit jedes Objekt einzeln zu beurteilen. Um jedoch Informationen über die Morphologie zu erhalten ist die rechts dargestellte Instanz-Segmentierung notwendig, welche algorithmisch am aufwendigsten zu implementieren ist.


Figure 1: Stained blood sample at a magnification of 400x, taken with a laboratory microscope (Keyence BZ 9000). Hemacolor® chemicals were used, which stain red blood cells violet and white blood cells dark violet.

An der Technischen Hochschule Wildau (TH Wildau) wurde in der Abteilung „Molekulare Biotechnologie und Funktionelle Genomik“ das Dissertationsprojekt Planktovision zur Identifikation verschiedener Phytoplankton-Spezies für die Beurteilung von Gewässerproben mithilfe automatischer Mikroskopie und Bilderkennung erfolgreich durchgeführt (Schulze et al., 2013). Hierbei wurde unmittelbar deutlich, dass ein mobiles und kostengünstiges Mikroskop, welches eine automatisierte Analyse vor Ort ermöglicht, deutliche Vorteile bieten würde. Um dieses Ziel zu erreichen, hat die Firma Oculyze, welche als Spin-off der TH Wildau erfolgreich ist, ein mobiles Smartphone-Mikroskop entwickelt (siehe Abbildung 2). Die Optik des Mikroskops ermöglicht eine 400fache Vergrößerung für Hellfeld-Untersuchungen. Die aufgenommenen Bilder werden direkt und automatisiert ausgewertet, sodass keine Fachkenntnisse erforderlich sind.
At the Technical University of Applied Sciences in Wildau (TUAS Wildau) in the division "Molecular Biotechnology and Functional Genomics" the PhD project Planktovision for the identification of different phytoplankton species to evaluate freshwater samples using automatic microscopy and image recognition was successfully carried out (Schulze et al., 2013). Herein it became obvious that a mobile, affordable microscope with automated analyses on site, would offer significant advantages. To achieve this goal, the company Oculyze, a successful spin-off of TUAS Wildau, has developed a mobile smartphone microscope (Figure 2). The optics of the microscope enables a magnification of 400x for brightfield examinations. Captured images can be evaluated directly and automatically without the need for specialist knowledge. In human diagnostics, the examination of the blood has high significance. A hematologist can quantify white and red blood cells, differentiate cell groups and/or detect morphological changes and draw conclusions about the general state of health and the immune system of a patient, examine deficiencies and diseases such as anemia and leukemia and detect blood parasites. The long-term goal of MoMiSmart (Mobile Microscopy on the Smartphone) - a subproject of digilog in TUAS Wildau works closely together with Oculyze GmbH - is to automatically detect and assess these indicators using a mobile microscope. The determination of the concentration and size of red blood cells, as well as a model to calculate the hematocrit content (Udroiu, 2014) have already been established using various image analysis algorithms. The results can be displayed directly in a smartphone application. In order to simplify handling in POCT, a movable sample holder was designed to evaluate as many cells of a blood sample as possible and to test the preparation of the blood samples using pre-colored slides.
In order to achieve the highest possible diagnostic accuracy, the complete image analysis will be based on DL algorithms in the future. Only the earlier mentioned instance segmentation permits a reliable analysis of cell morphology and type. This demanding task is performed in a cloud environment on an external server in order to enable the fastest possible evaluation. Due to the central availability of the data, their transfer to a specialist is also unproblematic in order to verify the result and/or to offer a therapy. Furthermore, a plan is in place to develop a specific hardware that can be used independent to the type of mobile device, not only to make the sample stage mobile, but to also motorize it and improve the optics for even better resolution and magnification. The functionality of this POCT system is shown in Figure 3.

mobile Mikroskop

Figure 3: The planned, mobile microscope is compatible with most android mobile phones and tablets via a USB connection. The microscopic images are transmitted encrypted to a cloud server and evaluated using powerful DL algorithms. The results are transferred directly to the mobile device or to a physician/hospital.


Kassenärztliche Bundesvereinigung. Ärztemangel. (24.10.18)
Kassenärztliche Bundesvereinigung. Mehr Pflegebedürftige in Regionen mit hohem Durchschnittsalter der Bevö
Pfeil J, Dangelat LN, Frohme M, Schulze K. Smartphone based mobile microscopy for diagnostics. The Journal of Cellular Biotechnology (in press)
Breslauer D, Maamari R, Switz N, Lam W, Fletcher D. Mobile phone based clinical microscopy for
global health applications. PloS one. 2009 Jul 22; 4(7):e6320.
Felsberg M. Five years after the Deep Learning revolution of computer vision: State of the art methods for online image and video analysis. Linköping University Electronic Press; 2017.
Schulze K, Tillich UM, Dandekar T, Frohme M (2013) PlanktoVision- an automated analysis system for the identification of phytoplankton. BMC Bioinformatics 2013 14:115.
Udroiu I. Estimation of erythrocyte surface area in mammals. arXiv preprint arXiv:1403.7660. 2014 Mar 29.

de_DEGerman en_GBEnglish