The Use of Large Language Models in Ophthalmology: A Scoping Review on Current Use-Cases and Considerations for Future Works in This Field

1. Introduction

The advancement and popularisation of generative artificial intelligence (AI) has resulted in its use permeating many areas of life and scientific research. This has largely been driven by the way Large Language Models (LLMs) have transformed the use of Natural Language Processing (NLP). Through self-supervised learning, LLMs have been utilised to effectively perform a wide variety of tasks ranging from interpreting and classifying text to generating answers to conversational questions. In November 2022, the release of ChatGPT by OpenAI revolutionised the LLM scene. Through its user-friendly interface and accessibility, ChatGPT has democratised the use of LLMs beyond the realm of computer science researchers, engaging a broad spectrum of users from various fields, sparking unprecedented interest in this field [1]. It took 4 years from the release of the Bidirectional Encoder Representations from Transformers (BERT) language model in October 2018 to develop 8 major LLM applications prior to ChatGPT’s release. On the contrary, in the two years since ChatGPT’s release, 8 major LLMs - Med-PaLM 1, Google Bard, Glass AI 2.0, GPT-4, Med-PaLM 2, LLaMa, Gemini and Claude were released (Figure 1).

Notably, newer LLMs have superior generalisation capabilities [1] and have been trained to provide more human-like responses, sparking interest in their use within medicine. To date, we have seen encouraging results supporting the use of LLMs in clinical practice, medical education and medical research [2,3,4,5,6].

The field of ophthalmology is no stranger to AI. Machine learning programs have been developed to detect and grade cataracts, while various deep learning programs have demonstrated their utility in detecting glaucomatous optic nerve changes. These applications have allowed ophthalmology to generate a wealth of data, paving the way for LLMs to potentially deliver more streamlined, personalised, and optimised care for ophthalmology patients [7,8,9].

It is unsurprising therefore, that amid this eruption of scientific output in the realm of AI and LLMs, a wide range of research regarding the usage and efficacy of LLMs in ophthalmology has emerged. Between January and July 2023 alone, a review summarising LLM trends in ophthalmology identified thirty-two articles related to this topic [10]. Inadvertently, this has also resulted in the publication of many isolated studies with overlapping scopes of research resulting in the duplication of efforts. In another review [11] of the usage of LLMs in ophthalmology, a total of 108 studies were identified between January 2016 to June 2023, 55 of which involved overlapping aspects of automated question-answering, while 27 dealt with information screening. Notably, this review did not provide a study-by-study breakdown of the 108 studies but mainly sought to understand general trends of LLM usage in ophthalmology. A literature review of publication in this field suggested that LLM research even in the niche area of ophthalmology appears to have a laissez-faire approach, with each having their own unique design. This potentially complicates the consolidation of research outputs in this field and makes it difficult to compare approaches and results across studies. To tackle such concerns, guidelines such as the SPIRIT-AI and CONSORT-AI initiatives for clinical trials and interventions involving AI have been created [12]. However, the extent to which such protocols are followed is yet to be determined. To our knowledge, to date there has also not been a summarisation of how LLM studies in the field of ophthalmology has been carried out.

In this study, we aim to map out the landscape of LLM applications in ophthalmology. By consolidating the work thus far, we aim to produce a point of reference to guide future research in this field. This review aims to summarise the following points:

• To identify recent studies (1st January 2019 – 11th February 2024) involving the application of LLMs in ophthalmology. This study period was chosen as it represents the period of LLM breakthroughs after BERT’s release [1] (Figure 1).
• To evaluate how studies of LLM applications in ophthalmology were carried out, in terms of following clinical trial protocols followed, prompt techniques employed, benchmarking methods used, and ethical considerations.
• To examine how LLMs fared in key areas of healthcare application, including exam taking and patient education, diagnostic and management capability, and clinical administration.
• To highlight potential issues surrounding the present landscape of LLM applications of ophthalmology, and to discuss directions for future LLM research and development in ophthalmology.

During our literature review, we found that studies utilising LLM in Ophthalmology covered a broad range of applications and had a diverse range of findings and methodologies. Given the broad and diverse nature of works in this field the format of a scoping review was chosen to map out the key trends and findings for this area in recent years, as opposed to a meta-analysis that seeks to draw a conclusion about specific research questions.

2. Methods

2.1. Search Strategy and Information Sources

A search of PubMed, Embase, SCOPUS, Web of Science, the Institute of Electrical and Electronics Engineers (IEEE) journals, the Association for Computing Machinery (ACM) journals, Google Scholar and DataBase systems and Logic Programming (DBLP) was performed from 1st January 2019 – 11th February 2024. The search strategy can be found in the supplementary information (Appendix Table A1).

The MeSH (medical subject heading) terms included are as follows:

1)

For Ophthalmology: Ophthalmology, Ocular Surgery, Eye Disease, Eye Diseases, Eye Disorders.

2)

For LLMs: Large Language Model, large language models, large language modelling, Chatbot, ChatGPT, GPT, chatbots, google bard, bing chat, BERT, RoBERTa, distilBERT, BART, MARIAN, llama, palm.

The search strategy was developed in consultation with expert opinion within the research team, which consisted of computer scientists (FXY,CCS) and clinicians [13] (KLZ (ophthalmology), SYKC (ophthalmology)). No additional search filters were applied.

2.2. Selection Process and Eligibility Criteria

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guideline was utilised [14]. An independent search was conducted by two reviewers (LKA, AWY). Any discrepancies were resolved through discussion between reviewers, with a third (KLZ) included when necessary. This review has not been registered prior.

The inclusion criteria were

1)

Peer reviewed primary research studies utilising LLMs.

2)

Studies involving ophthalmology.

3)

Studies published from January 2019 to March 2024.

The exclusion criteria were

1)

Study designs that were reviews, systematic reviews and meta-analyses, case reports, case series, guidelines, letters, correspondences, or protocols.

2)

Studies that were not published in English.

2.3. Data Extraction and Analysis

Data on the studies were uploaded into Mendeley and imported into COVIDENCE Systematic Review Software (Veritas Health Innovation, Melbourne, Australia) for screening. As mentioned earlier, differences in screening outcomes were resolved in consultation with a third reviewer.

Data extracted from the papers were analysed on Microsoft Excel (Microsoft, Richmond, Virginia, USA). These included (1) authorship details, (2) LLMs utilised, (3) study methodology, and (4) performance and performance scoring of the LLMs.

In terms of study methodology, we took note of clinical trial protocols used, prompt techniques employed, how benchmarking was done, and ethical considerations in the studies.

The performance of LLMs were also analysed in the following areas: exam taking and patient education, diagnostic capability, management capability, clinical administration, inaccuracies, and harm (glossary A).

The subspecialties studied included (1) Cornea, (2) Glaucoma, (3) Neuro-ophthalmology, (4) Uveitis, (5) Lens and cataract, (6) Paediatrics and strabismus. (7) Retina and Vitreous, (8) Oculoplastics, (9) Optics, (10) Refractive surgery and (11) Pathology. The various LLMs were then assessed on their accuracy and overall completeness of their answers, which were then ranked and compared across the different LLMs employed per study.

3. Results

A total of 976 studies were screened, for which 904 were excluded with 72 being sought for retrieval. A further 22 of these studies did not meet the inclusion criteria, and a final 49 studies [15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63] were included in this study (Figure 2, Table 1).

3.1. Overall Study Characteristics

A total of 14 different LLMs models (Table 1) were used across the studies with various applications spanning the fields of administration, clinical knowledge, diagnostics, exam taking, manuscript writing, patient education, prognostication, text interpretation, and triage. GPT-3.5 was the most commonly employed LLM, being utilised in 34 studies. GPT-4.0 came in second, appearing in 26 studies. Bard and Bing LLM models were the next most used (Table 1).

In terms of study design, all studies did not follow a standardised clinical trial protocol for artificial intelligence. All studies employed a zero-shot, one-shot or few-shot prompt engineering technique, apart from one study which only utilised contextual priming. There were three studies who additionally used prompt chaining, iterative prompting, and chain-of-thought prompt techniques to supplement their work. Most studies (37 of 49 studies) shared full examples of their prompts (Table 2). Across the studies, the grading of the output generated by the LLMs was heterogeneous with little standardisation, resulting in difficulty in data analyses. 24 studies employed human assessors to benchmark LLM performance in terms of “correctness” of output, two of which were assisted by automated benchmarking assessments. The remaining studies utilised automated benchmarking to assess “correctness” of output. Nine studies considered harm in their study protocol, all of which were assessed by humans. Only one study by Wilhelm et. al. also utilised an automated form of harm assessment in the form of GPT-4.0 Twelve studies delved into the ethical implications relating to their work, while thirteen only touched very briefly on patient safety without going further into an ethical discussion (Table 2).

3.2. Breakdown of LLM Benchmarks Studied and General Observations

43 of the 49 included studies [15,16,17,18,19,20,21,22,23,24,25,26,27,29,30,31,32,34,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,56,57,58,59,60,62,63] performed an assessment regarding the “correctness” of the LLM’s output in some form - be it in the form of relevance or accuracy, to name a few. The remaining 6 studies looked at other qualities such as readability [28,35,36], manuscript writing [33], and administration [43,55]. Amongst the 43 studies assessing the “correctness” of the LLM’s output, 27 [16,17,18,20,23,27,29,30,31,32,34,37,39,41,45,47,49,50,51,52,54,56,57,59,60,62,63] of them compared multiple LLMs against each other (Table 3a), while 16 [15,19,21,22,24,25,26,38,40,42,43,44,46,48,53,58] were observational studies using a single LLM (Table 3b). A total of eleven different scoring systems were used to assess for “correctness” (Appendix Table A2). There were 15 studies [20,23,30,31,34,37,41,47,49,51,54,56,57,59,63] which compared GPT-4.0 against humans and/or other LLMs. Among these, GPT-4.0 was the best performer in 10 studies [20,30,31,34,37,41,47,57,59,63]. Amongst the seven studies [16,27,29,37,47,52,56] comparing Bard, Bing, and GPT-3.5, GPT-3.5 had the best performance in 5 [16,29,37,47,56] of them. Amongst the single-armed studies, the LLMs were reported to have largely appropriate responses overall (Table 3b). In the following subsections, we go into further detail regarding LLM performance in specific domains.

Human vs Artificial Intelligence

16 studies [17,21,23,25,26,30,31,38,39,41,49,50,51,53,56,57] investigated the performance of LLMs against humans (Attendings, Ophthalmologists-in-training) in diagnosis, exam taking, patient education, prognostication, and triage. In terms of diagnostic, prognostic, and triage ability, humans consistently outperformed LLMs in all six of these studies (Table 4). In terms of answering exam questions, there was a more even balance with humans being the best in three studies, while GPT-4.0 superseding humans in four studies. It is also worth noting that humans consistently performed better than GPT-3.5 in exam taking for every subspecialty (Table 5). The same could be said in terms of developing patient education materials, with humans bettering GPT 3.5 in one study, equalling GPT 3.5 in another study, and partnering with AI to supersede GPT-4.0 in the last study of this area. Notably, the latter study found that GPT 3.5 produced superior results to humans in terms of how empathetic their patient education material was. (Table 4)

3.3. Performance of LLM in Exam-Taking and Patient Education

Of the 14 studies which assessed LLM exam-taking capabilities, ten performed focused analysis of individual ophthalmology subspecialties (Table 4). GPT-4.0 was consistently the top performing LLM in all these studies, also scoring more than 50% of answers correct in all but one study (Table 5). The study where GPT-4.0 scored less than 50% of answers correct was performed using a Japanese question bank, highlighting the possible language barriers inherent to LLMs (Table 5).

20 studies looked at patient education with 16 assessing performance and relevance of output (8 comparative [16,29,37,45,47,56,62,63], 8 non-comparative [19,21,22,24,38,42,46,58]) and the remaining 4 assessing readability [28,35,36,61] (Table 1). Amongst the eight comparative studies, GPT-4.0 was deemed to produce the best patient educational materials in three of the four studies [37,47,63] that it was involved in, while GPT-3.5 performed the best in the remaining four studies [16,29,45,62]. Looking at non-comparative studies, it was found that only GPT-3.5 and 4.0 were used. Both models performed well with the majority of responses being “good”, scoring more than 50%, or assessed as “relevant”, depending on the scoring systems applied (Table 3b, appendix 1). Regarding readability of patient educational materials, a total of nine different scoring systems were used amongst the 4 studies, showing how varied assessment in this area can be. Results here varied greatly even within individual studies depending on the types of prompts given (Table 6). Both Bard and GPT-4.0 were able to significantly improve the readability scores by varying the types of prompts given [28,35]. GPT-3.5 performed inconsistently with the material produced being beyond the desired reading level in 1 study [35] and being at the desired reading level in another study [36]. In Eid’s study [28], GPT-4.0 generated material that was easier to read than Bard. Meanwhile, without prompts, Bard was able to provide educational material that was easier to read in Eid’s study [28] but not Kianian’s [35].

3.4. Diagnostic and Management Capabilities of LLM

11 studies [23,26,37,38,39,48,49,50,53,57,63] assessed the diagnostic capabilities of LLM through cases and questions. GPT-4.0 consistently outperformed other LLMs (BingChat, WebMD, Bard) in coming to a diagnosis. (Table 7). As mentioned earlier, humans still performed better than LLMs in this field, nevertheless most studies did not report a great disparity between LLM score versus human scores. It was noted that most LLM outputs included a cautionary line such as “you should seek medical attention from a medical professional”.

7 studies [16,22,23,24,37,50,57] evaluated the management of eye conditions, 5 of which employed multiple LLMs for comparison [16,23,37,50,57]. As noted earlier in LLM diagnostic ability, GPT-4.0 was also superior to GPT-3.5 and Bard in suggesting the appropriate management (Table 8). 2 studies compared the performance of LLMs against humans here [23,57]. The performance of GPT-4 was found to be better or similar to humans in both studies (Table 8). Again, it was noted that most LLMs included a cautionary disclaimer to seek professional medical advice.

3.5. Clinical Administration Tasks

Only 3 studies evaluated the use of LLM for clinical administration tasks [33,43,55]. In two of the studies which gave LLMs more freedom to write, significant levels of hallucinations were observed.

1 study looked at using LLMs for discharge summary and operative notes writing [55]. It found that the quality of GPT’s discharge summaries were affected by the quality of the prompts and tended to be valid but generic. Here, GPT-3.5 hallucinated its own model of the intraocular lenses utilised, but when prompted further, it was able to self-correct to improve the quality of output (Table 9).

Another study [33] evaluated manuscript abstract writing using GPT-3.5 and GPT-4.0. GPT-4.0 outperformed its predecessor on all fronts, including DISCERN score, helpfulness, truthfulness, and harmlessness. However, it was noted that both versions had hallucinated references (Table 9).

The last study was more focused, testing LLM on classifying texts into retina international classification of diseases (ICD) coding [43]. Of the 181 prompts given, 70% of the prompts had at least one correct ICD code generated by the LLM. This accuracy was reduced to 59% when assessed to generate only the correct ICD code (Table 9).

3.6. LLM Inaccuracies and Harm

20 studies [15,16,19,20,21,22,23,24,29,33,37,38,39,40,42,46,47,56,60,62]detailed the hallucinations or inaccuracies produced by the LLMs. Bard demonstrated a significant inaccuracy rate, having the most inaccuracies in 4 of the 6 studies it was involved in [16,29,37,47,56,62]. On the other hand, GPT-4.0 had the lowest inaccuracy rate amongst LLMs in all 7 of the studies which included inaccuracy analysis and it [20,23,33,37,39,47,56]. In single LLM studies [15,19,21,22,24,38,40,42,46], we observed that inappropriate responses made up a minority of responses and were at times comparable to the frequency of errors in human answers (Table 10).

11 studies [19,20,21,24,33,37,40,46,56,60,63] evaluated the potential for, the extent of, and the likelihood of harm by the LLMs. In comparative studies [20,24,33,37,40,56,60,63], GPT-4.0 was less likely to generate harmful content when compared to GPT-3.5, Claude 2, Bing and Bard. In some studies, GPT-4.0 did not generate responses that constituted harm [19]. Only two studies compared harm from LLMs against that of humans, both of which found that likelihood of harm by humans and LLM were equivalent [21,56]. Extent of harm was equivalent between humans and chatbots in the study by Bernstein et al [21], while this was lowest in humans in the study by Tailor et al [56] (Table 11).

4. Discussion

This scoping review identified a total of 49 primary research studies applying LLMs in ophthalmology that were published in the five years and two months’ time period of the search. These studies explored a wide range of applications, thereby providing breadth to this nascent field. The results of this scoping review suggest that while state-of-the-art LLMs can exhibit human-level performance, their real-world clinical application still faces several challenges. In the subsequent sections we discuss our results in the context of the study objectives and the implications for evidence and future research. Firstly, we evaluate the conduct of LLM studies in the field of ophthalmology. Thereafter we examine the performance of LLMs in ophthalmology based on current research, according to the major domains of their current applications – namely: patient education and exam taking, ophthalmic diagnostic capabilities, management capabilities, and clinical applications. We then discuss the existing drawbacks and hurdles facing the use of LLMs in ophthalmology. Finally, we discuss directions for future LLM research and development in ophthalmology. To our knowledge, this is the first such review to provide analysis and critique on the conduct of research in the field of LLMs and ophthalmology.

4.1. Evaluation of Past Methodologies

4.1.1. Issues Regarding Standardisation

Amidst the excitement to gather data regarding LLM applications, we have found that recent publications have not been seen to follow suggested frameworks or protocols. Rather, we see diverse pockets of data being collected by individual studies over multiple fields. While there is utility in this for widening the breadth of the data pool, the lack of standardized benchmarks leads researchers and experts to use varying benchmarks and implementations, resulting in inconsistent and sometimes incomparable evaluation results. We noted that all included studies also did not follow a fixed AI-related research protocol. This hampers the ability of follow up studies in reproducing these precedents. In the same vein, 12 of the 49 studies did not provide full examples of their prompts, potentially affecting reproducibility of their works. Following protocolised guidelines for AI related clinical trials, the open sharing of specific prompt techniques employed, and the usage of common benchmarks allows for research works in the realm of LLM to be more reproducible and suitable for direct comparison. The SPIRIT-AI and CONSORT-AI initiatives for clinical trials and interventions involving AI are examples of such protocols. Taking the SPIRIT-AI extension for example, interventions are required to specify the procedure for acquiring and selecting the input data for the AI intervention, and to specify the procedure for assessing and handling poor quality or unavailable input data [64]. Such accountability and transparency of steps would benefit future works seeking to build on previous research and allow for better comparison of results.

Beyond issues regarding transparency and standardisation, we noted inconsistencies in terms of the benchmarking of LLM performance. In our study, we encountered significant heterogeneity with respect to the grading systems with some studies grading on a Likert scale with 1 being the worst and 5 being the best [56] and others with 1 being the best and 3 being the worst [29]. Similarly, in evaluating diagnostic capabilities, scoring systems could be binary, meaning whether the responses were correct or incorrect [57], while other studies evaluated agreement with experts [50]. This hinders the inability to perform statistical analysis across studies and hence limits future meta-analysis in this field. Another source of inconsistency was the use of human evaluation. While human evaluation is necessary to grade areas such as harm, many of such evaluations appeared to be arbitrary and not based on evidence-based grading criteria. It is heartening to see open-source frameworks for benchmarking medical machine learning models such as MedPerf gaining traction, but these are yet to be widely adopted [65].

Most studies appeared to take the first output from their LLM platforms. Potential irreproducibility of answers from LLM platforms is a known fact. Answers generated on one occasion may differ from answers generated upon subsequent inputs of the same question. Singer et.al sought to overcome this by considering only the initial answers generated [54], however this runs the risk of missing out better or worse answers subsequently. Future works can seek to overcome this by taking the average of multiple outputs from their LLM platform, such as a best-of-three format.

4.1.2. Harm and Patient Safety

There was a general lack of consideration for patient harm, being evaluated in only nine studies and ethics only being formally discussed in 12 of the included studies (Table 2). As medicine is by nature a practice of non-maleficence, the objective of “doing no harm” has been central to clinical trials throughout medicine. Clinical trials employing AI should be no different, and aids in keeping the patient’s welfare at the heart of everything. While many of the included studies were not direct applications on real patients per se, the limited attention paid to ethical safeguards serve as a timely reminder for the future as LLM applications assimilate further with medical practice. Despite being primarily evaluation of technology and not having live patient involvement in most of the studies, the real-world implications of these studies is undeniable. It would be useful for future works to state their findings in relation to patient safety – for instance the certainty to which GPT-4.0 could provide reliable medical advice within a specific field of ophthalmology. The Assessment List for Trustworthy AI (ALTAI) is an example of a self-assessment checklist published by the European Commission as an ethics guideline for trustworthy AI in July 2020 [66,67]. Checklists such as these could be included as supplementary material in AI studies relating to healthcare, serving as an ethical safeguard for patients.

Of the nine studies which evaluated the harm of LLM output, only two studies compared this to harm from human output [21,56]. While data on harm ought to be retrieved from LLMs, it would be insightful when such output is taken in relation to harm from human output. By obtaining human data in the same context for a basis of comparison, we can understand if LLM output is truly more harmful or would a human expert in the same confines of the study be any better. It is worth noting as well that the evaluation of harm involved human assessment in all nine studies in which it was evaluated, showing how LLMs still require a human safety net at this point (Table 2).

4.1.3. Other Issues Relating to Study Design

Amongst the included studies, there also appears to be multiple studies of similar design. Many of these employ zero-shot prompts to test the capabilities of LLMs in a particular area, and then assess their accuracy via exact match benchmarking, or via human assessment. The utility of such repetitive studies overlapping in design is questionable.

There was also an overwhelming bias towards using GPT-3.5 and GPT-4.0, making up the overwhelming majority of LLMs used in the included studies (Table 1). The benefit of doing so is the deeper exploration of GPT models, which are reportedly the most popular LLMs in use in recent history [68]. On the other hand, this runs the risk of under-representing other LLM models. Hence, while current research may provide a good indication of GPT applications in ophthalmology, it may not be representative of LLMs as a whole.

4.2. Evaluation of LLM Performance

Broadly speaking, all the included studies explored two main areas of “correctness” and “inaccuracies”, while a subset also studied the readability and harm of the LLM’s output. In studies where GPT-4.0 was included, it was amongst the best-performing LLMs in all domains of patient education and exam taking, ophthalmic diagnostic capabilities, management capabilities, and clinical applications.

In exam taking ability, LLM could equal and even surpass human scores. Even when faced with inductive subspecialties such as neuro-ophthalmology, GPT-4.0 could perform to the level of or even better than humans [41,57]. It is worth noting however, that all exam questions were text based. Two studies [34,49] attempted to assess the medical image reading ability of LLMs. However, these studies did so by using text-based descriptions of the images as input, rather than a raw image itself. It is known that LLMs have the ability to analyse images, and testing its ability to analyse raw medical image files directly would pave the way for further clinical utility within ophthalmology. Such attempts have already been carried out by non-ophthalmology based studies [69], but results are inconclusive at this point. We also identified an instance where GPT-4.0 was uncharacteristically poor. When a Japanese question bank was utilised, GPT-4.0 performed the worst, scoring less than 50%. Similarly, English prompts fared better than Chinese prompts when reporting fundus fluorescein angiography reports using GPT-3.5 [38]. Possible reasons for this could be the language difference, which GPT-4.0 might not have had exposure to or been trained on and hence fared poorer. This highlights a potentially inherent weakness in LLMs, whereby performance could be hindered by a lack of exposure to the language. While English is a dominant language globally, it is estimated to be spoken by only 20% of the world’s population [70]. The lack of multilingual support is a potential barrier that future works may consider exploring further.

In the area of patient education, it is perhaps unsurprising to find that expert edited LLM responses fared the best in terms of quality [56]. Tailor et. al. reported that human expert-edited LLM responses performed better than purely human expert responses and saved more time when compared to the experts creating a response from scratch [56]. Similarly, Bernstein's study comparing LLMs with those of ophthalmologists found comparable quality in the advice provided [21]. These works demonstrate an interesting direction where more effective human-AI collaboration might be achieved – an area underexplored by most studies which tended to benchmark pure LLM output on its own without human revision. On the other hand, a surprising finding was that LLM output could exceed that of humans in terms of empathy scores [56]. This isolated finding was another underexplored yet highly relevant area in this field, as healthcare is not merely a practice of knowledge, but also an art that requires the humanitarian touch. Lastly, the ability of LLMs to personalize the readability of patient education materials to their audience’s comprehension levels strengthens its position for future adoption as demonstrated by Bard [28,63].

In terms of diagnostic and management capabilities, LLMs appeared to struggle more when coming to a diagnosis [26,49,50] but fared better when asked for the management plan after the diagnosis had been established. This reflects the higher order thinking that is required for making a diagnosis. In the study by Rojas-Carabali et al., we note that LLMs were possibly disadvantaged in that they were given text descriptions of images, while humans were given the images to assess [50]. It would be useful for future works to assess how LLMs would perform against humans if both were given the same images to come to a diagnosis. It has been shown that a simple combination of patient history and chief complaint could predict an overall diagnostic accuracy of approximately 90% of neuro-ophthalmology cases when read by human assessors [71]. These results seem to suggest that the ability of LLMs to interpret written information falls short of humans despite their potentially greater wealth of knowledge. Also as pointed out earlier, most LLMs included medical disclaimers when posed with diagnostic questions (e.g. “you should seek medical attention from a medical professional”). This drives home the point that while LLMs may close the gap on human accuracy in diagnosis, there is still some way to go before their opinion is taken to be as legitimate as that of a medical professional.

The area of clinical administration tasks was only covered by three studies touching different areas. The dearth of data here calls for more work to explore this area of untapped potential. Singh's research highlights ChatGPT's ability to swiftly generate detailed ophthalmic discharge summaries and operative notes [55], showcasing its potential to streamline administrative processes with tailored content and rapid response times. Similarly, Ong's study demonstrates ChatGPT's capability to interpret text accurately [43], suggesting its potential to ease physician burden in tasks like ICD coding. Moreover, Hua's investigation [33] into manuscript writing reveals that AI-generated ophthalmic scientific abstracts are comparable in quality between different versions of GPT-3.5 and GPT-4.0, though factual errors in references indicate a need for further refinement. Overall, these findings show that LLMs can be helpful for administrative tasks in ophthalmology, but more work is needed to establish them further for practical use in healthcare and ophthalmology.

4.3. Directions for Future Works

4.3.1. Standard Framework for Assessing Accuracy, Validity and Harm

Much like other reviews and commentaries on LLMs in other fields [72,73], this study calls for future works to follow standardised benchmarks and frameworks for assessing the accuracy and validity of LLMs in clinical settings [74]. A robust framework would offer clear guidelines in areas such as providing comprehensive context on diseases, precise wording reporting, incorporating diverse question formats, adopting learning techniques, and using standardised metrics like ROUGE (Recall-Oriented Understudy for Gisting Evaluation) and METEOR (Metric for Evaluation of Translation with Explicit ORdering) [73]. The Standardized Assessment Framework for Evaluations of Large Language Models in Medicine (SAFE-LLM) is one such model which sets out to unify evaluation standards, facilitating the comparison and improvement of LLMs in medical applications [75]. Such frameworks establish a common language and understanding between developers and end-users, fostering collaboration and partnership in the advancement and deployment of LLMs. However, as seen in this review they are yet to be widely adopted. Medical image processing is a growing field within AI, and comprehensive benchmarks such as MedSegBench are already being developed in this area [76]. This is especially relevant to ophthalmology – a field highly dependent on image interpretation. Moving forward it is imperative that standardised benchmarks in this area are employed as well.

Clinical trial protocols such as CONSORT-AI and SPIRIT-AI emphasise the importance of describing the results of any performance errors and how errors were identified. Conducting future LLM studies in line with such protocols would address the critical need for transparency and accountability in assessing the safety and reliability of LLMs, further contributing to building trust among developers and end-users. With standardised AI study protocols in place, stakeholders can communicate their findings with more transparency and uniformity, ensuring the ethical and responsible use of LLMs in various domains, including healthcare. Many of the current works of LLMs in ophthalmology are not yet in the clinical trial stage with live patient testing. This could be due to the ethical considerations mentioned in the next section. Nevertheless, defined protocols even for such studies can build the foundation of transparency and accountability for future real-world patient applications.

4.3.2. Greater Evaluation and Strategies Towards Ethical Considerations

It is encouraging that considerations and suggestions regarding medical ethics were raised in some of the studies in this review. We highlight some of them here.

The inaccuracies of LLM output raises the risk of harm to patients. Al-Sharif et al suggested that LLMs be trained solely on supervised evidence-based ophthalmology datasets, to maintain the “purity” of what the LLM “knows” [16]. Singer [54] and Antaki's [18] studies emphasise the importance of using verified sources to ensure the trustworthiness and accuracy of information provided by LLMs. It has also been shown that fine-tuning medical LLMs significantly improves their safety, reducing their tendency to comply with harmful requests [77]. LLM as a tool has the potential to do good and harm, and it is the responsibility of LLM creators and clinicians to ensure that they are developed to adequate safety standards to limit the harm on patients.

Despite the ability of LLMs to have a reasonably high rate of accuracy, LLM inaccuracies may still be interspersed amongst these facts. Bernstein et al highlighted that these partial truths in LLM outputs may lure patients into a false sense of trust [21]. As shown previously, medical disclaimers are frequently used at the end of LLM medical outputs to mitigate this. Raghu et al also highlighted that equally important is the education of end-users about the capabilities, potential risks, and benefits of this technology [48].

Bernstein et al also noted that patient healthcare information would have to be entered into LLMs to obtain customised and individualised output. OpenAI’s privacy policy states that they “may collect Personal Information that is included in the input” [78]. Patient data entering the online domain, or into the servers of private companies are at risk of being hacked and has implications for patient confidentiality and data privacy [79]. In the clinical deployment of LLMs, policies should include strategies to safeguard this. Raghu et al suggested that until such safeguards are in place, only anonymised patient data should be entered into these LLMs [48], while Tao et al suggested to keep personalisation of output offline after online drafts are generated [58].

Regarding information source, there lies the issue of plagiarism as brought up by Tao et al [58]. Data authenticity, data provenance and intellectual property contamination are issues that LLMs are still grappling with [79]. Text generated from LLM output may be taken from copywrite sources illegitimately. We have also seen cases of LLMs hallucinating references [33]. To date, LLM reliability for citation and reference have been found to be inconsistent and occasionally very poor [80]. Further fine-tuning of LLMs in this regard should be a priority moving forward, whilst end-users ought to query the original sources and cite where credit is due.

Jiao et al raised the issue of biases inherent to LLMs, which risks amplifying existing health disparities. LLMs may refer to source material that does not represent all patient populations equally resulting in unequal treatment for specific patient groups [34]. We have also seen how GPT underperformed when tested in a non-English language [51], potentially underserving patients who speak non-English languages. Utilising adversarial testing and bias detection algorithms to identify and remove any discriminatory patterns in the prompts or the AI-generated outputs are possible ways to tackle these biases [81]. While training LLMs on diverse and representative sources are possible ways to reduce inequalities associated with LLM use, Kianian et al also argues that improving readability of LLM output can reduce such inequalities too [36]. This is because with poor readability comes poorer health literacy, which have been show to disproportionately affect populations of lower socioeconomic status [82]. Collaboration between prompt engineers, bioethicists and patient advocates may help in designing prompts that are inclusive, diverse, and free from biases based on factors such as race, ethnicity, gender, or socioeconomic status [81].

Finally, Tao et al also questioned how the burden of legal responsibility should be divided between physician and LLM, especially for cases of patient harm or privacy breaches [58]. As AI systems become increasingly autonomous and capable of decision making, it is important to ensure that there is accountability for their actions. This includes ensuring that AI systems are transparent and that there are oversight mechanisms in place to address any errors [66].

4.3.3. Techniques for Improving LLM’s Accuracy and Interpretability

In general, prompt engineering, a transformative approach in natural language processing, involves the development of tailored input prompts or instructions to guide LLMs in generating desired outputs or responses. Examples of such methodologies include Retrieval-Augmented Generation (RAG) and fine-tuning. Fine-tuning involves adjusting the model's parameters based on task-specific datasets, essentially operating in a "close-book" manner. Conversely, RAG functions in an "open book" setting, harnessing external information sources to retrieve and integrate relevant data, thereby enhancing the model's comprehension and generative capabilities. For instance, in the domain of healthcare education [83], RAG was chosen due to its capability to provide traceable responses, enhancing trust and explainability, its scalability in accessing vast healthcare knowledge bases, and its flexibility for rapid updates in alignment with evolving clinical guidelines.

Some studies, as seen in patient education use-cases, improved LLM performance by innovative prompt engineering and fine-tuning. This suggests that the limiting factor of output may not only be in the LLM itself, but rather the types of prompts given. Further works exploring the effect of using varying styles of prompts on LLM output would aid in verifying this. Both Eid [28] and Kianian [35] improved the readability of their patient education material output by specifying a level of reading (the 6th-grade reading level in their case). Lim et al. found that even by using a simple prompt “That does not seem quite right. Could you kindly review?”, GPT-3.5, GPT-4.0 and Bard were able to able to demonstrate substantial self-correction abilities [37]. Bernstein et al. used instruction prompt engineering to answer patient’s questions. This prompt technique uses explicit instructions or cues about the task at hand to adapt the behaviour of the LLM model [21]. With the use of these prompts, they found that human-written and AI-generated answers to patient ophthalmology-related questions were very comparable in terms of accuracy and harm. Notably, assessors could not be “definitely sure” if the responses were AI or human generated in the majority of cases. Another study by Liu et al. [38] utilised chain-of-thought-inspired prompt techniques to elucidate a step-by-step reasoning process from GPT-3.5 for both English and Chinese prompts. Interestingly this study found that English prompts performed better for diagnostic and inference capabilities, as well as providing more complete reasoning steps, suggesting that choice of language affects the quality of output as well.

4.4. Strengths and Limitations

The strengths of this review include the wide search strategy, involving eight bibliographic databases involving both the fields of medical and information technology. The time frame chosen as part of the search criteria (2019 - 2024), gives a reflection of the scene of LLM usage within ophthalmology in this period of LLM breakthroughs since the release of BERT in October 2018. This review also followed best practices in the PRISMA-ScR for conducting a scoping review [14,84]. Expert opinions in the fields of LLM and ophthalmology were also consulted. This was in line with best practice recommendations by the Institute of Medicine (US) Committee on Standards for Systematic Reviews of Comparative Effectiveness Research [85], as well as Arksey and O’Malley’s and Levac et al.’s frameworks for scoping reviews [13,86].

To our knowledge, this is the first scoping review to critique the methodology and conduct of LLM research in ophthalmology. Based on these findings describing the current landscape of LLM research in ophthalmology, this study puts forth key recommendations to strengthen the lack of standardisation and ethical regulation amongst LLM-related studies, and tangible steps to improve the conduct of future works in this field.

Nevertheless, there were shortcomings with regards to the conduct of this review. The search terms chosen aimed to capture all studies relating to LLMs and ophthalmology within the given timeframe. However, due to the rapidly evolving nature of LLMs, newer yet relevant search terms may inadvertently be missed out on. The use of MeSH terms was done with the aim of improving reproducibility of results. However, this ran the risk of missing out on recent articles not yet indexed. The strict exclusion criteria on study design also sought to improve the quality of evidence collected in this review. Nonetheless, this also runs the risk of missing out on novel data, such as from case reports, which is especially possible in the growing field of LLMs. As a trending and growing field, advancements in LLMs are rapid, and recent developments are bound the be missed. For instance, promising and relevant LLM models such as DeepSeek were not covered in any of the included studies. Constant attempts to update the paper to chase each new publication hampers the progress of the paper. Overall, we believe that we have captured a significant portion of time and publications to represent this field at a time when interest in LLMs skyrocketed, while allowing the thoughtful evaluation and discussion of our findings.

The heterogeneity of measures employed in assessing LLMs, and the wide range of study designs made it difficult to compare findings across studies, and to provide firm conclusions. We therefore sought to summarise the assessment of LLMs by the various studies by placing these evaluations into the overarching categories of “exam taking and patient education”, “diagnostic capability”, “management capability”, “clinical administration”, and “inaccuracies and harm”. Many of the included studies utilised subjective modes of assessment that lacked in strength of evidence, for instance in determining degree of “correctness” or frequency of hallucinations and nonlogical reasoning. Nevertheless, such studies were included as this review did not discriminate against studies based on the strength of their study design, and to reflect the current climate of how LLMs are assessed.

5. Conclusion

LLMs have received considerable attention through their introduction to the general public and have found potential applications in the field of medicine, and in particular, ophthalmology. The main use cases are in exam-taking, patient education, diagnosis and management, and clinical administration. We presented an overview of the landscape of LLM applications in ophthalmology. In our study, we found that the majority of LLMs perform acceptably, with GPT-4.0 having one of the best performances. However, issues pertaining to hallucination, inaccuracies, and harm still exist. We also evaluated how past studies of LLMs in ophthalmology have been carried out and summarized their findings. We have also identified gaps in current literature and have made suggestions for future works to improve on, with hopes that future works can form a more cohesive and clinically useful pool of knowledge that can be applied to patients in a safe and ethical manner. We conclude by advocating for the adoption of standardised frameworks to assess LLMs in healthcare and recommend techniques to improve the performance of LLM in niche fields such as ophthalmology.