mlsanityData to 14 Oct 2026

Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs

arXiv 2412.14304 ↗First posted 18 Dec 2024cs.CL cs.AIcited by 10

On the map: Retrieval-augmented generation → Medical and biomedical RAG

Abstract (arXiv)

Current ophthalmology clinical workflows are plagued by over-referrals, long waits, and complex and heterogeneous medical records. Large language models (LLMs) present a promising solution to automate various procedures such as triaging, preliminary tests like visual acuity assessment, and report summaries. However, LLMs have demonstrated significantly varied performance across different languages in natural language question-answering tasks, potentially exacerbating healthcare disparities in Low and Middle-Income Countries (LMICs). This study introduces the first multilingual ophthalmological question-answering benchmark with manually curated questions parallel across languages, allowing for direct cross-lingual comparisons. Our evaluation of 6 popular LLMs across 7 different languages reveals substantial bias across different languages, highlighting risks for clinical deployment of LLMs in LMICs. Existing debiasing methods such as Translation Chain-of-Thought or Retrieval-augmented generation (RAG) by themselves fall short of closing this performance gap, often failing to improve performance across all languages and lacking specificity for the medical domain. To address this issue, We propose CLARA (Cross-Lingual Reflective Agentic system), a novel inference time de-biasing method leveraging retrieval augmented generation and self-verification. Our approach not only improves performance across all languages but also significantly reduces the multilingual bias gap, facilitating equitable LLM application across the globe.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. Builds on 32 of its 61 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

32 of 61 references.

References · 32

  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  3. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  4. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  5. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 333
  6. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  7. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  8. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 165
  9. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 95
  10. 2023Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Datasetcited by 91
  11. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 84
  12. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
20 more
  1. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedbackcited by 51
  2. 2024xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoningcited by 46
  3. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 39
  4. 2024Gender Bias in Large Language Models across Multiple Languagescited by 35
  5. 2024Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMscited by 33
  6. 2024Bias patterns in the application of LLMs for clinical decision support: A comprehensive studycited by 27
  7. 2024Can Large Language Models Understand Context?cited by 26
  8. 2023FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domaincited by 25
  9. 2024Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustnesscited by 22
  10. 2024Prompting Fairness: Integrating Causality to Debias Large Language Modelscited by 21
  11. 2024Spanish and LLM Benchmarks: is MMLU Lost in Translation?cited by 20
  12. 2024Multi-FAct: Assessing Factuality of Multilingual LLMs using FActScorecited by 18
  13. 2024Healthcare Copilot: Eliciting the Power of General LLMs for Medical Consultationcited by 17
  14. 2024Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generationcited by 17
  15. 2024Enhancing Retrieval and Managing Retrieval: A Four-Module Synergy for Improved Quality and Efficiency in RAG Systemscited by 15
  16. 2024Breaking the Language Barrier: Can Direct Inference Outperform Pre-Translation in Multilingual LLM Applications?cited by 12
  17. 2023Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizercited by 7
  18. 2024Automated Clinical Data Extraction with Knowledge Conditioned LLMscited by 5
  19. 2023Can LLMs Grade Short-Answer Reading Comprehension Questions : An Empirical Study with a Novel Datasetcited by 4
  20. 2024From Generalist to Specialist: Improving Large Language Models for Medical Physics Using ARCoTcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025MedPlan: A Two-Stage RAG-Based System for Personalized Medical Plan Generationcited by 5
  2. 2025Surfacing Subtle Stereotypes: A Multilingual, Debate-Oriented Evaluation of Modern LLMscited by 3
  3. 2025On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AIcited by 2
  4. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  5. 2025Zoom In Disparities in Healthcare LLM Q&Acited by 1
  6. 2026Beyond English benchmarks: clinical llm evaluation in Brazilian Portuguesecited by 1
  7. 2025Insight-A: Attribution-aware for Multimodal Misinformation Detectioncited by 0
  8. 2026Measuring Political Stance and Consistency in Large Language Modelscited by 0
  9. 2026Reliable Multilingual Orthopedic Decision Support from Clinical Narratives: Language-Aware Adaptation and Verification-Guided Deferralcited by 0
  10. 2026Perspectives on Cross-Lingual Consistency in LLMs for Medical Questionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking Next-Generation Reasoning-Focused Large Language Models in Ophthalmology: A Head-to-Head Evaluation on 5,888 Items
  2. 2025BEnchmarking LLMs for Ophthalmology (BELO) for Ophthalmological Knowledge and Reasoning
  3. 2024Enhancing Large Language Models with Domain-specific Retrieval Augment Generation: A Case Study on Long-form Consumer Health Question Answering in Ophthalmology
  4. 2025Benchmarking Large Multimodal Models for Ophthalmic Visual Question Answering with OphthalWeChat
  5. 2025Performance of GPT-5 Frontier Models in Ophthalmology Question Answering
  6. 2026EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding
  7. 2024LEME: Open Large Language Models for Ophthalmology with Advanced Reasoning and Clinical Validation
  8. 2024Development and bilingual evaluation of Japanese medical large language model within reasonably low computational resources