# Evaluating multiple large language models in pediatric ophthalmology arXiv: https://arxiv.org/abs/2311.04368 ยท HTML: https://mlsanity.com/paper/2311.04368/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Builds on 36 of its 63 references that are in the map. - First posted on arXiv 7 Nov 2023. ## Builds on (most cited first, 30 of 36) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022) - [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019) - [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021) - [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019) - [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022) - [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019) - [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023) - [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020) - [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023) - [Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://mlsanity.com/paper/2304.01852/) (2023) - [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021) - [AugGPT: Leveraging ChatGPT for Text Data Augmentation](https://mlsanity.com/paper/2302.13007/) (2023) - [DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4](https://mlsanity.com/paper/2303.11032/) (2023) - [ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models](https://mlsanity.com/paper/2302.07257/) (2023) - [Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations](https://mlsanity.com/paper/2303.18027/) (2023) - [When Brain-inspired AI Meets AGI](https://mlsanity.com/paper/2303.15935/) (2023) - [Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics](https://mlsanity.com/paper/2304.01938/) (2023) - [SAMAug: Point Prompt Augmentation for Segment Anything Model](https://mlsanity.com/paper/2307.01187/) (2023) - [AD-AutoGPT: An Autonomous GPT for Alzheimer's Disease Infodemiology](https://mlsanity.com/paper/2306.10095/) (2023) - [OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialogue](https://mlsanity.com/paper/2306.12174/) (2023) - [Detecting LLM-Generated Text in Computing Education: A Comparative Study for ChatGPT Cases](https://mlsanity.com/paper/2307.07411/) (2023) - [Differentiate ChatGPT-generated and Human-written Medical Texts](https://mlsanity.com/paper/2304.11567/) (2023) - [MKRAG: Medical Knowledge Retrieval Augmented Generation for Medical Question Answering](https://mlsanity.com/paper/2309.16035/) (2023) - [Exploring New Frontiers in Agricultural NLP: Investigating the Potential of Large Language Models for Food Applications](https://mlsanity.com/paper/2306.11892/) (2023) ## Built on by (most cited first, 4 of 4) - [Large Language Models for Manufacturing](https://mlsanity.com/paper/2410.21418/) (2024) - [Towards Next-Generation Medical Agent: How o1 is Reshaping Decision-Making in Medical Scenarios](https://mlsanity.com/paper/2411.14461/) (2024) - [Large Language Models for Bioinformatics](https://mlsanity.com/paper/2501.06271/) (2025) - [Affective-ROPTester: Capability and Bias Analysis of LLMs in Predicting Retinopathy of Prematurity](https://mlsanity.com/paper/2507.05816/) (2025) ## Abstract (arXiv) IMPORTANCE The response effectiveness of different large language models (LLMs) and various individuals, including medical students, graduate students, and practicing physicians, in pediatric ophthalmology consultations, has not been clearly established yet. OBJECTIVE Design a 100-question exam based on pediatric ophthalmology to evaluate the performance of LLMs in highly specialized scenarios and compare them with the performance of medical students and physicians at different levels. DESIGN, SETTING, AND PARTICIPANTS This survey study assessed three LLMs, namely ChatGPT (GPT-3.5), GPT-4, and PaLM2, were assessed alongside three human cohorts: medical students, postgraduate students, and attending physicians, in their ability to answer questions related to pediatric ophthalmology. It was conducted by administering questionnaires in the form of test papers through the LLM network interface, with the valuable participation of volunteers. MAIN OUTCOMES AND MEASURES Mean scores of LLM and humans on 100 multiple-choice questions, as well as the answer stability, correlation, and response confidence of each LLM. RESULTS GPT-4 performed comparably to attending physicians, while ChatGPT (GPT-3.5) and PaLM2 outperformed medical students but slightly trailed behind postgraduate students. Furthermore, GPT-4 exhibited greater stability and confidence when responding to inquiries compared to ChatGPT (GPT-3.5) and PaLM2. CONCLUSIONS AND RELEVANCE Our results underscore the potential for LLMs to provide medical assistance in pediatric ophthalmology and suggest significant capacity to guide the education of medical students.