Prerpints.org logo
Preprint
Article

Real World Analysis of Two Handheld Retinograph, Evaluation by Ophthalmologist and an Artificial Intelligence Algorithm

Altmetrics

Downloads

79

Views

42

Comments

0

Submitted:

10 September 2024

Posted:

10 September 2024

You are already at the latest version

Alerts
Abstract
(1) Telemedicine in diabetic retinopathy (RD) screening is effective but does not reach the entire diabetes population. The use of portable cameras and artificial intelligence (AI) can help diabetes screening. (2) Methods: We evaluated the ability of two handheld cameras, one based on a smartphone and the other on a smartscope, to obtain images comparing OCT. Evaluation was done in two stages, the first by two retina specialists and the second using an artificial intelligence algorithm that we developed. (3) Results: The retina specialists report that the smartphone images must mydriasis in all cases compared to 73.05% of the smartscope images and 71.11% of the OCT images. Images were ungradable in 27.98% of the retinographs with the smartphone and 7.98% with the smartscope. The detection of any-DR using the AI algorithm showed that the smartphone obtained lower recall values (0.89) and F1scores (0.89) than the smartscope, with 0.99. The detection of Mild-DR using the smartphone also obtained low results (146 retinographs) compared to the smartscope (218 retinographs). (4) Conclusions: The use of handheld devices together with AI algorithms for reading retinographs can be useful for DR screening, although these devices need to improve the ease image acquisition through small pupils.
Keywords: 
Subject: Medicine and Pharmacology  -   Ophthalmology

1. Introduction

Diabetic retinopathy (DR) is a complication of diabetes mellitus (DM) that affects retinal vasculature in the form of microangiopathy and at the same time the neurons that compose it in the form of neuropathy, which means that diabetic retinopathy. Current it is considered to be a neurovascular involvement of the retina [1].
DM currently affects 537 million patients worldwide and is predicted to rise to 643 million by 2030 and to 783 million by 2045 [2]. The increase in DR runs parallel, currently affecting 6.4 million people in Europe alone with an expected increase to 8.6 million by 2025, of whom 30% will need treatment [3]. DR also remains the leading cause of preventable vision loss and blindness in adults aged 20 to 74 years, especially in middle- and high-income countries [4,5].
Given the rapid global increase in DM and increasing life expectancy, DR will remain a major public health challenge. Screening for DR through telemedicine has proven to be effective [6] therefore promoting it where it is already in place and implementing it where it is not. Only early detection of DR will allow us to prevent its evolution to more severe forms and to treat it early to avoid vision loss [7].
Current screening systems can reach a significant proportion of the patients with DM, but it is difficult to screen all patients every one or two years, as recommended [2]. In our experience, only 40% of patients are screened annually [8], despite having four non-mydriatic camera units in our health area. To improve this, extending screening to two years according to the metabolic control of patients has been proposed, along with bringing in other professionals, such as general practitioners (GPs), endocrinologists or pediatricians into the system. At a technical level, the use of portable cameras has been proposed that can be used by GPs, together with artificial intelligence (AI) algorithms that read retinographs automatically and help to detect DR [9,10].
Based on these proposals, the present study evaluated two portable cameras and an AI algorithm to detect DR developed by our team, and the results were compared [11].

2. Materials and Methods

2.1. Design

A study based on a real population of patients with type 2 DM. The reference being our Health Area (Hospital Universitari de Sant Joan de Reus, Spain) of 226,508 inhabitants, in which 17,792 patients are registered with DM. The study has been ongoing since 2007, when we began offering annual retinography to our patients with type 2 DM. The full screening programme is described in full elsewhere [12].
Briefly, screening is carried out from retinographs obtained by non-mydriatic cameras. Images are stored in the computerized medical record and are later read by specialists (general practitioners, endocrinologists or pediatricians). We have recently replaced three of the four non-mydriatic cameras with optical coherence tomography (Triton OCT™) (hereafter referred to as Triton) equipment, which allows us to detect patients with macular pathology.

2.2. Objectives

The first was to evaluate the ability of two portable cameras to take good quality retinographs. Images were captured with an Android smartphone (VOLK™ Vistaview, Topcon Healthcare®) (hereafter referred to as Vistaview) and with a smartscope whose images are similar to the classic retinograph (Aurora™, Optomed Topcon Healthcare®) (hereafter referred to as Aurora).
The first step was to evaluate the images captured from the devices. Two independent retina specialists observed the images from the Triton taken by the non-mydriatic camera unit located at the Hospital Universitari Sant Joan de Reus, where the study was carried out. This was considered the gold standard when determining the concordance of the results between teams.
The second step was to evaluate the retinographs obtained from the two portable systems and those obtained from the Triton using an AI algorithm that we have developed, called MIRA© [11]. Briefly, the MIRA© algorithm was developed from the public database MESSIDOR© [13] and our MIRADATASET© database [14] which contains 40692 retinograms labelled according to whether or not they have DR and what type of RD, using the MESSIDOR classification that classifies DR into three levels [15].

2.3. Inclusion Criteria

Patients with type 2 DM from our DR screening programme.

2.4. Exclusion Criteria

Type 1 DM and other causes of diabetes mellitus such as gestational DM, and retinographs that were not in a patient's electronic medical record or did not have DICOM format.

2.5. Methods

Retinographs were obtained randomly from patients presenting for DR screening. For each patient, a retinograph of each eye was obtained from the three devices, the Vistaview, the Aurora and the Triton. The images were focused on a point located between the fovea and the temporal side of the papilla, with retinographs being 45⁰-55⁰. Pharmacological mydriasis was achieved by applying tropicamide eye drops if necessary. The retinographs were sent via the PACS (Picture Archiving and Communication System) system to the patient's electronic medical record (EMR), where they were stored in DICOM (Digital Imaging and Communications in Medicine), a standard format that guarantees the interoperability and compatibility of medical images.
Table 1 shows DR classified with the international classification of DR [16] at 'level 1, or mild-DR', and 'level 2, or moderate-DR'. At 'level 3, or severe-DR' we introduced a variant to include images with newvessels in the retina (proliferative DR) and we included any patients with macular edema. Therefore, our level 3 would be equivalent to a vision-affecting referrable DR or VTDR (vision threatening diabetic retinopathy).

2.6. Technical Characteristics of the Three Devices

Table 2 shows the variations between the three cameras used to obtain retinographs. The Vistaview is a portable retina camera that incorporates an Android smartphone as a capture system that allows the examination of the retina with a field of view of 55⁰.
The Aurora is a smartscope, a handheld retina camera, which allows examination of the retina with a field of view of 50⁰, covering the macula and papilla.
The Triton is a scan-sourced optical coherence tomograph device with a scan speed of 100 kHz and a wavelength of 1,050 nm, which delivers a true-colour, non-mydriatic fundus image.

2.6. Ethics and Consent

The study was carried out with the approval of the local ethics committee, the Medical Research Ethics Committee (CEIM) of the Pere Virgili Health Research Institute (IISPV), Tarragona, Spain, RetinaReadRisk approval code, protocol version 1. 10/03/2022, CEIM reference number: 071/2022

2.7. Statistical Methods

Data was analysed via the SPSS program, version 22.0 (IBM® Statistics, Chicago, IL, USA). A descriptive statistical analysis of the quantitative data was performed by determining the mean and standard deviation. For qualitative data, frequency and percentage analysis in each category were used. For all parameters, descriptive statistics (including number of values, mean, standard deviation (SD) and standard error of mean (SEM) were calculated. A p value less than 0.05 was considered statistically significant.
The screening performance of the AI Machine Reading Algorithm (MIRA) © was measured using a 2x2 confusion matrix/contingency table for each team. Given a classified dataset, there were four basic combinations of real and assigned: true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN).
Statistical evaluation of the dataset included sensitivity, specificity, positive predictive value or accuracy, negative predictive value, harmonic mean or F1 score, and accuracy.
Sensitivity or recall is the proportion of the population for which the test is correct and, in clinical diagnosis, sensitivity and specificity values are considered good when they exceed 80%. Positive or accuracy predictive value, and negative predictive value, give a proportion of the population with a given test result for which the test is correct and incorrect, respectively.
To assess concordance, we determined the values of accuracy and harmonic mean. Diagnostic accuracy is expressed as the proportion of correctly classified subjects among all subjects, accuracy values of >0.8 show an almost perfect correlation, while values of 0.6-0.79 imply a significant correlation, 0.4-0.59 a moderate correlation, 0.2-0.39 a regular correlation and 0-0.2 a poor correlation. The harmonic mean or F1 score, also known as the Sørensen-Dice coefficient or Dice similarity coefficient (DSC), is a measure of test performance that combines precision (the positive predictive value) and recall (sensitivity), the highest possible value of F1 is 1, indicating perfect accuracy and recall, and the lowest possible value is 0 if the accuracy or recall is zero. The advantage of using the harmonic mean over accuracy is that it is not affected by the prevalence of the disease, which is the case with accuracy, so if the prevalence is low it tends to give high accuracy values [17].

3. Results

3.1.

A retinograph was taken of each eye with each of the three devices. In total, 4260 retinographs were taken of 2130 patients. The mean age of the sample was 67.43±11.21 years (38-91 years), of which 1256 (58.96%) were women. Regarding the treatment of DM, 264 patients (12.39%) were treated through diet alone, 1810 (84.98%) with oral antidiabetics and 56 (2.63%) with insulin or insulin + oral antidiabetics. Finally, 1527 (71.69%) had high blood pressure.
Table 3 shows the differences between the retinographs across the three devices. We first tried to take the retinographs without pupil dilatation, but the images were without DR in 69.83% with the Vistaview, in 83.91% of the cases with the Aurora, and 71.11% of cases with the Triton.
After dilation, the percentage of images that were not of sufficient quality for ophthalmologists to evaluate was 27.98% with the Vistaview, 7.98% with the Aurora but only 0.93% of the images obtained with Triton. The principle causes of poor quality were
(I) the difficulty in focusing the handheld device, particularly the Vistaview, (ii) the presence of cataracts, obviously affecting all three devices equally and (iii) Little mydriasis
Patients with moderate-DR or severe-DR were identified with similar success across all three devices, but for mild-DR the Aurora and the Triton were more successful than the Vistaview.

3.2. Statistical Analyses

Data comparing the handheld devices with the Triton is presented in Table 4. Applying the 2x2 contingency table we found the two handheld devices to be reliable. Although the Vistaview has a somewhat lower score in the application of the F1 statistic or harmonic mean (0.93 for any-DR and 0.91 for Mild-DR), the results were good for both in any case.

3.3. The Comparative Study of the Results of the Three Devices Using the MIRA© Algorithm

Table 5 shows the results from reading the images obtained with the MIRA© automatic reading AI algorithm.
According to the type of DR, the MIRA© algorithm reads retinographs from the Aurora and Triton devices more accurately than those from the Vistaview. Mild-DR is detected in fewer cases with the Vistaview (102 cases) compared to the Aurora (218) and the Triton (220). The latter are what we consider real cases since they were identified by retina specialists and is the gold standard. The AI algorithm identified moderate-DR and severe-DR with similar success across the three devices.

Statistical Study of the Results Obtained by AI Algorithm MIRA©

We evaluated the images obtained by the three devices, the Triton tomography being what we currently consider to be the gold standard, with images read by specialist ophthalmologists.
Table 6 shows the statistical analysis of the performance of the three devices in the detection of DR, using 2x2 contingency tables. The sensitivity of the Vistaview smartphone to identify any-DR was 0.89, being outperformed by the portable Aurora (0.99) and the images obtained by the Triton OCT(0.99)
Similarly, for Mild-DR, the scores for the Vistaview drop to a sensitivity of 0.83 with a positive predictive value of 0.85 compared to the values of 0.99 sensitivity for both the Aurora and the Triton and a positive predictive value value of 0.98 for the Aurora. This is confirmed by the TP values and by the F1 harmonic average score, the smartphone scoring a harmonic average of 0.89 for detecting any-DR and 0.83 for mild-DR.
In summary, the statistical study indicates that the Aurora and Triton OCT devices detect with high reliability when DR is present or absent at all stages of DR. The Vistaview device is safe when it indicates that there is no DR but unreliable in detecting cases of Mild-DR.

4. Discussion

The constant increase and diversity of the population with DM has led to the pursuit of new ways of reaching as many of the diabetic population as possible and various publications by scientific societies have reported progress via telemedicine [2,6].
The authors believe that portable devices used together with AI algorithms that can read retinographs will be able to reach a larger population than the current model allows [18,19,20,21,22,23].
The present study aimed to evaluate the usefulness of two of those portable devices in screening patients with diabetes mellitus. We selected one smartphone camera, and a smartscope. We compared the images obtained with those two devices with the Triton optical coherence tomography camera, which is the standard for DR screening in our health area. Results show that the smartphone produced a higher number of ungradable retinographs (27.98%) than the Aurora smartscope (7.98%) or the standard Triton OCT (0.93%). We can report that mydriasis was necessary for all patients when using the smartphone compared to 73.05% using the Aurora and 71.11% using the Triton. The major difficulty was being able to focus accurately on the retina when using the smartphone. We can also report that the smartphone had more difficulty in distinguishing mild-DR forms (only 93 cases) than the Aurora (109 cases) and the Triton (110 cases).
In the second part, we analyzed the results after the automatic reading of retinogtraphies using our MIRA© algorithm. As in the first part of the study, the automatic image analysis detects far fewer cases of retinographies with Mild-DR in the images obtained with the Vistaview device compared to the other two devices, thus only detecting 146 retinographies with Mild-DR compared to 218 with the Aurora or 220 with the OCT. We could conclude that somehow for both the human eye and the AI algorithm the detection of Mild-DR is more difficult with the smartphone, perhaps the difference in pixelation of the images obtained by the smartphone which is 28.4 Mpixels compared to 5 Mpixels for the other two devices (Table 2).
Although the images obtained with all three devices could be considered good, the statistical analysis revealed that the sensitivity or recall, and the positive predictive value or precision, are poorer for the images obtained with the smartphone, both in terms of any-DR or Mild-DR. The differences were less marked when using the harmonic mean or F1 score, which scored the Vistaview smartphone at 0.97 for accuracy and the Aurora smartscope at 0.99 for the presence of any-DR or mild-DR. However, when applying the harmonic average the values change, at 0.93 (any-DR) and 0.91 (mild-DR) for the Vistaview and 0.98 and 0.95, respectively, for the Aurora. These differences are due to what we have commented on methods, accuracy is affected by prevalence and cases such as the one in this study with RD prevalences of less than 9% make the determination of the harmonic mean as a statistic to differentiate cases more accurate.
A similar study by Jacoba et al [24] was a clinical series on 225 eyes of 116 patients in which the presence of macular pathology alone was evaluated. That study used two portable retina smartscopes, the Aurora and the RetinaVue™, comparing them with the Cirrus™ OCT camera. The study reported good specificity but low sensitivity of the portables in detecting maculopathies.
Another similar study aimed not only to screen for DR but also to detect macular pathology, whether diabetic or not [23]. The study reported ungradabilty with the OCT at 0.9%, a figure similar to ours, and 4.4% with the Aurora, a value somewhat lower than ours, which is 7.98%, although the focus was on the macular area rather than on the whole retina.
A review of the efficacy of smartphones via a meta-analysis carried out by Tan et al [21] reported on five Android or iPhone5S mobile devices. Results showed that for the detection of any-DR mean sensitivity was 87% (minimum 74%, maximum 94%) and specificity was 94% (minimum 81%, maximum 98%), values not dissimilar to those we obtained using the Vistaview (sensitivity of 88% and specificity of 99%), although notably sensitivity for mild-DR wasvery low with an average of 39% (minimum 10% maximum 79%). It is important to take into account that smartphone technology is constantly changing and improving [25].
The Optomed™ Aurora has been tested in various studies and results have been similar to ours. The largest sample is that of Salongcay et al. In 2023 [26]. That study reported a percentage of unreadable images of 7.5%, similar to ours with 7.07%, and an agreement of 82.4%, lower than ours at 98%. Other studies have reported similar values to ours, with growth figures of 93% in Kubin et al. [27] and a positive predictive value of 98% and a specificity of 97% in the Salongcay et al. [28].
One of the strengths of the present study is that the analysis of images has been carried out from those found in the electronic medical record (EMR) rather than those stored in the equipment. That brings together the images in DICOM format for all devices and allows us to compare them more easily. Another strength is the much larger sample size compared to most previous studies. We were able to include 4260 images from 2130 patients, and only the Salongcay study of 2023 includes a higher number with 5585 images from 2793 patients.A limitations of the study is clearly having used only two devices, which makes extrapolation of the results to other devices unsafe. The use of a single retinograph focused on a point between the macula and temporal side of the papilla might also limit the detection of DR against studies that include more retinographs of more fields or perhaps wide field retinographs, which have been shown to change the severity of DR [29]. Another point to note is that at level 3 of DR we included patients with proliferative retinopathy and macular edema, which might alter the results when compared to other studies, although it can be used as a reference for studies that want to evaluate the number of patients we have detected with DR-referable.

5. Conclusions

We can affirm that image capture systems or handheld retina cameras are useful for detecting DR in screening programmes and might be further helped by incorporating AI algorithms. We have found images taken with a smartscope or portable retinograph device better than with a smartphone, which gave us a large number of unreadable images. Both devices require more efficient focusing, in order to make image centering easier, improve image readability, and improve the quality of images taken through small pupils to 3 mm, reducing the need for mydriasis.

6. Patents

Software MIRA, register SAFE CREATIVE code 2007104712196. Date 10 de julio de 2020 a las 11:24 UTC

Author Contributions

Conceptualization, P.R-A., B. F-P., E.G-C., A.V., J.C., I. M-M., and M. B-B; methodology, P.R-A., B. F-P. E.G-C., A.V., M.LL-S.,; C.M-L.,; I. M-M., and M. B-B; Validation, P.R.-A., E.G-C., M.B.-B., A.V., J.C., M.LL-S.,; C.M-L.,; and A.M.; formal analysis, P.R-A., E.G-C., A.I-B., A.V., J.C., M.LL-S.,; C.M-L.,; and B.F-P.; investigation, P. R-A., E.G-C., M.B-B., A.V., and J.C., resources, P.R-A., B. F-P.. E.G-C., J M A.V., I. M-M., and M.B-B; writing—original draft preparation, P.R-A., E.G-C., A.V. M. B-B., and B.F-P., .; writing—review and editing, P.R-A., B. F-P. E.G-C., A.V., J.C., M.LL-S.,; C.M-L.,; I. M-M., and M. B-F-P;, supervision, P.R.-A., A.V., J.C., B.F-P., R.V., I. M-M., and M. B-B.; project administration, P.R-A., E.G-C., M.B.-B., A.V., and B.F.funding acquisition, P.R-A., A.V., I. M-M., and M. B-B;. All authors have read and agreed to the published version of the manuscript.

Funding

This project has received co-funding from the European Institute of Innovation and Technology (EIT) Health and the European Union under Grant Agreements 220718 and 230123 and from the Instituto de Salud Carlos III (ISCIII) through the project PI21/00064 and co-funded by the European Union.

Institutional Review Board Statement

The study was conducted in accordance with the Decla-ration of Helsinki and approved by the Institutional Review Ethics Committee (CEIM) of Institut d'Investigació Sanitaria Pere Virgili (IISPV), Tarragona, Spain, approval code RetinaReadRisk, protocol version 1. 10/03/2022, Refer-ence number CEIM: 071/2022.

Informed Consent Statement

Not applicable.

Data Availability Statement

Statement: the original contributions presented in the study are included in the article, further enquiries can be directed to the corresponding authors.

Acknowledgments

We thank endocrinologists in our hospital for helping us to implement the new screening system using the non-mydriatic fundus cameras and our camera technicians for their work and interest in the diabetes screening. We thank all the patients for being participants in the present study. We also thank Phil Hoddy for his language assistance and for editing and correcting the English text.

Conflicts of Interest

The authors declare no conflicts of interest. The funders had no role in the design of the study, or in the collection, analyses or interpretation of the data, in the writing of the manuscript, or in the decision to publish the results.

References

  1. Simó R, Frontoni S. Neuropathic damage in the diabetic eye: clinical implications. Curr Opin Pharmacol. 2020 Dec; 55:1-7. [CrossRef]
  2. IDF Diabetes Atlas, 10th edn. Brussles, Belgium: International Diabetes Federation; 2021. Disponible en: https://www.diabetesatlas.org. (accessed 2024-07-24).
  3. Li JQ, Welchowski T, Schmid M, et al. Prevalence, incidence and future projection of diabetic eye disease in Europe: a systematic review and meta-analysis. Eur J Epidemiol. 2020 Jan;35(1):11-23. [CrossRef]
  4. Leasher JL, Bourne RR, Flaxman SR,et al; Vision Loss Expert Group of the Global Burden of Disease Study. Global Estimates on the Number of People Blind or Visually Impaired by Diabetic Retinopathy: A Meta-analysis From 1990 to 2010. Diabetes Care. 2016 Sep;39(9):1643-9. [CrossRef]
  5. Vision Loss Expert Group of the Global Burden of Disease Study; GBD 2019 Blindness and Vision Impairment Collaborators. Global estimates on the number of people blind or visually impaired by diabetic retinopathy: a meta-analysis from 2000 to 2020. Eye (Lond). 2024 Aug;38(11):2047-2057. [CrossRef]
  6. Weng CY, Maguire MG, Flaxel CJ, et al. Effectiveness of Conventional Digital Fundus Photography-Based Teleretinal Screening for Diabetic Retinopathy and Diabetic Macular Edema: A Report by the American Academy of Ophthalmology. Ophthalmology. 2024 Aug;131(8):927-942. [CrossRef]
  7. Hainsworth DP, Bebu I, Aiello LP, et al; Diabetes Control and Complications Trial (DCCT)/Epidemiology of Diabetes Interventions and Complications (EDIC) Research Group. Risk Factors for Retinopathy in Type 1 Diabetes: The DCCT/EDIC Study. Diabetes Care. 2019 May;42(5):875-882. [CrossRef]
  8. Romero-Aroca P, López-Galvez M, Martinez-Brocca MA, et al. Changes in the Epidemiology of Diabetic Retinopathy in Spain: A Systematic Review and Meta-Analysis. Healthcare (Basel). 2022 Jul 16;10(7):1318. [CrossRef]
  9. Grzybowski A, Rao DP, Brona P, et al. Diagnostic Accuracy of Automated Diabetic Retinopathy Image Assessment Softwares: IDx-DR and Medios Artificial Intelligence. Ophthalmic Res. 2023;66(1):1286-1292. [CrossRef]
  10. Vought R, Vought V, Shah M, et al. EyeArt artificial intelligence analysis of diabetic retinopathy in retinal screening events. Int Ophthalmol. 2023 Dec;43(12):4851-4859. [CrossRef]
  11. Baget-Bernaldiz M, Pedro RA, Santos-Blanco E, et al. Testing a Deep Learning Algorithm for Detection of Diabetic Retinopathy in a Spanish Diabetic Population and with MESSIDOR Database. Diagnostics (Basel). 2021 Jul 31;11(8):1385. [CrossRef]
  12. Romero-Aroca, P.; De La Riva-Fernandez, S.; Valls-Mateu, A.; et al. Changes observed in diabetic retinopathy. Eight years follow up of a Spanish population. Br. J. Ophthalmol. 2016, 100, 1366–1371.
  13. https://www.adcis.net/en/third-party/messidor/ (accessed 2024-07-24).
  14. RetinaReadRisk (RRR). Available at: https://www.iispv.cat/cas-dxit/retinareadrisk-rrr/ (Date Accessed: 2024-07-24).
  15. Romero-Aroca P, Verges-Puig R, de la Torre J, et al. Validation of a Deep Learning Algorithm for Diabetic Retinopathy. Telemed J E Health. 2020 Aug;26(8):1001-1009. [CrossRef]
  16. Wilkinson CP, Ferris F, Klein RE, et al. Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology 2003, 110, 1677–1682.
  17. Shreffler J, Huecker MR. Diagnostic Testing Accuracy: Sensitivity, Specificity, Predictive Values and Likelihood Ratios.[Updated 2023 Mar 6]. InStatPearls [Internet]; StatPearls Publishing: Treasure Island, FL, USA, 2024. Available online: https://www.ncbi.nlm.nih.gov/books/NBK557491/ (accessed on 6 August 2024).
  18. Weng CY, Maguire MG, Flaxel CJ, et al. Effectiveness of Conventional Digital Fundus Photography-Based Teleretinal Screening for Diabetic Retinopathy and Diabetic Macular Edema: A Report by the American Academy of Ophthalmology. Ophthalmology. 2024 Aug;131(8):927-942. [CrossRef]
  19. Grauslund, J. Diabetic retinopathy screening in the emerging era of artificial intelligence. Diabetologia. 2022 Sep;65(9):1415-1423. [CrossRef]
  20. Jacoba CMP, Doan D, Salongcay RP, et al. Performance of Automated Machine Learning for Diabetic Retinopathy Image Classification from Multi-field Handheld Retinal Images. Ophthalmol Retina. 2023 Aug;7(8):703-712. [CrossRef]
  21. Tan CH, Kyaw BM, Smith H, et al. Use of Smartphones to Detect Diabetic Retinopathy: Scoping Review and Meta-Analysis of Diagnostic Test Accuracy Studies. J Med Internet Res. 2020 May 15;22(5):e16658. [CrossRef]
  22. Palermo BJ, D'Amico SL, Kim BY, Brady CJ. Sensitivity and specificity of handheld fundus cameras for eye disease: A systematic review and pooled analysis. Surv Ophthalmol. 2022 Sep-Oct;67(5):1531-1539. [CrossRef]
  23. Kubin AM, Huhtinen P, Ohtonen P, et al. Comparison of 21 artificial intelligence algorithms in automated diabetic retinopathy screening using handheld fundus camera. Ann Med. 2024 Dec;56(1):2352018. [CrossRef]
  24. Jacoba CMP, Salongcay RP, Rageh AK, et al. Comparisons of Handheld Retinal Imaging with Optical Coherence Tomography for the Identification of Macular Pathology in Patients with Diabetes. Ophthalmic Res. 2023;66(1):903-912. [CrossRef]
  25. de Oliveira JAE, Nakayama LF, Zago Ribeiro L, et al. Clinical validation of a smartphone-based retinal camera for diabetic retinopathy screening. Acta Diabetol. 2023 Aug;60(8):1075-1081. [CrossRef]
  26. Salongcay RP, Aquino LAC, Alog GP, et al. Accuracy of Integrated Artificial Intelligence Grading Using Handheld Retinal Imaging in a Community Diabetic Eye Screening Program. Ophthalmol Sci. 2023 Dec 15;4(3):100457. [CrossRef]
  27. Kubin AM, Wirkkala J, Keskitalo A, et al. Handheld fundus camera performance, image quality and outcomes of diabetic retinopathy grading in a pilot screening study. Acta Ophthalmol. 2021 Dec;99(8):e1415-e1420. [CrossRef]
  28. Salongcay RP, Aquino LAC, Salva CMG, et al. Comparison of Handheld Retinal Imaging with ETDRS 7-Standard Field Photography for Diabetic Retinopathy and Diabetic Macular Edema. Ophthalmol Retina. 2022 Jul;6(7):548-556. [CrossRef]
  29. Jacoba CMP, Salongcay RP, Aquino LAC, et al. Comparisons of handheld retinal imaging devices with ultrawide field images for determining diabetic retinopathy severity. Acta Ophthalmol. 2023 Sep;101(6):670-678. [CrossRef]
Table 1. Classification used in this study. Level 3 includes severe-DR, proliferative DR, and macular edema. DR= diabetic retinopathy, AMD = diabetic macular edema.
Table 1. Classification used in this study. Level 3 includes severe-DR, proliferative DR, and macular edema. DR= diabetic retinopathy, AMD = diabetic macular edema.
Level Description
0 No retinopathy.
1 Mild-DR. Microaneurysms only.
2 Moderate-DR. More than just microaneurysms, but less than severe.
3
  • Severe-DR. Any of the following: more than 20 intraretinal haemorrhages in the 4 quadrants; venous beads defined in two quadrants; Intraretinal microvascular abnormalities in one quadrant.
  • Proliferative retinopathy.
  • If DME is present will included in Level 3.
Levels of DME included in level 3 of current study Mild DME: Some thickening of the retina or hard exudates at the posterior pole but distant from the centre of the macula.
Moderate DME: thickening of the retina or hard exudates that approach the centre of the macula but do not involve the centre.
Severe DME: thickening of the retina or hard exudates affecting the centre of the macula.
Table 2. Characteristics of the three devices used.
Table 2. Characteristics of the three devices used.
Vistaview Aurora Triton
Field 55⁰ 50⁰ 50⁰
Pupil 4 mm 3.5 mm 3.5 mm
Image Resolution 28.4M pixels 5 Mpixels 5M pixels
Fountain smartphone Rechargeable lithium-ion battery Mains
Mydriasis Mydriatic Non-mydriatic/mydriatic Non-mydriatic / Mydriatic
Table 3. Comparative study of the two portable devices against the evaluation by two retina specialists. %* percentage of the total number of patients with diabetic retinopathy.
Table 3. Comparative study of the two portable devices against the evaluation by two retina specialists. %* percentage of the total number of patients with diabetic retinopathy.
Patients Total retinographs Patients who required mydriasis Ungradable retinographs
Vistaview 2130 4260 All cases 1192 / 27.98%
Aurora 2130 4260 1556 /73.05% 340 / 7.98%
Triton 2130 4260 1536 / 71.11% 20 / 0.93%
No DR Any DR Mild-DR Moderate- RD Severe- RD
Vistaview 1962 / 69.83% 168 /7.88% 93/ 55.37%* 57/ 33.92%* 18/10.71%*
Aurora 1948 / 83.91% 186 / 8.54% 109/ 58.39%* 59/ 31.72%* 18/ 9.89%*
Triton 1948 / 90.03% 188 / 8.82% 110 / 58.51%* 60 / 31.91%* 18 / 9.57%*
Table 4. Study of patients with and without DR. TN true negative, FN false negative, TP true positive, FP false positive, S sensitivity, SP specificity, TPV true positive value, NPV negative predictive value, F1 score or mean harmony and ACC accuracy.
Table 4. Study of patients with and without DR. TN true negative, FN false negative, TP true positive, FP false positive, S sensitivity, SP specificity, TPV true positive value, NPV negative predictive value, F1 score or mean harmony and ACC accuracy.
TN FN TP FP S SP POS NPV ACC F1score
Any-DR Vistaview 1400 20 168 24 0.89 0.98 0.99 0.99 0.97 0.93
Aurora 1942 2 188 6 0.99 0.99 0.97 0.99 0.99 0.98
TN FN TP FP S SP POS NPV ACC F1score
Mild-DR Vistaview 1404 17 93 20 0.85 0.99 0.82 0.99 0.93 0.91
Aurora 1942 1 109 8 0.99 0.99 0.93 0.99 0.99 0.95
Table 5. The analysis of the images entered into the MIRA© automatic reading algorithm.
Table 5. The analysis of the images entered into the MIRA© automatic reading algorithm.
Classification of retinographs
Ungradable or MIRA No DR Any DR Mild-DR Moderate- DR Severe- DR
Vistaview 1362 / 31.97% 2330/54.46% 284 /7.88% 146/ 55.37% 102/ 33.92% 36/10.71%
Aurora 342 / 8.02% 3174/74.5% 372 / 8.54% 218/ 58.39% 118/ 31.72% 36/ 9.89%
Triton 40 / 0.93% 3468/81.4% 376 / 8.82% 220/ 58.52% 120/ 31.91% 36/9.57%
Table 6. Statistical analysis by AI algorithm. TN true negative, FN false negative, TP true positive, FP false positive, S sensitivity, SP specificity, TPV true positive value, NPV negative predictive value, F1 score or mean harmony and ACC accuracy.
Table 6. Statistical analysis by AI algorithm. TN true negative, FN false negative, TP true positive, FP false positive, S sensitivity, SP specificity, TPV true positive value, NPV negative predictive value, F1 score or mean harmony and ACC accuracy.
TN FN TP FP S SP POS NPV ACC F1score
Any-DR Vistaview® 2614 30 284 34 0.89 0.99 0.90 0.99 0.97 0.89
Aurora® 1201 2 130 1 0.99 0.99 0.98 0.99 0.99 0.98
Triton 3842 2 376 2 0.99 0.99 0.99 0.99 0.99 0.99
TN FN TP FP S SP POS NPV ACC F1score
Mild-DR Vistaview® 2614 25 146 30 0.83 0.99 0.85 0.99 0.98 0.83
Aurora® 1201 2 130 1 0.99 0.99 0.98 0.99 0.99 0.98
Triton 3842 2 220 2 0.99 0.99 0.99 0.99 0.99 0.99
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
Prerpints.org logo

Preprints.org is a free preprint server supported by MDPI in Basel, Switzerland.

facebook logotwitter logolinkedin logo
MDPI Initiatives
Important Links
Subscribe

Disclaimer

Terms of Use

Privacy Policy

© 2024 MDPI (Basel, Switzerland) unless otherwise stated