# Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs

arXiv: https://arxiv.org/abs/2412.14304 · HTML: https://mlsanity.com/paper/2412.14304/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 32 of its 61 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Placed in

- Retrieval-augmented generation / [Medical and biomedical RAG](https://mlsanity.com/rag/medical-and-biomedical-rag/)

## Builds on (most cited first, 30 of 32)

- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents](https://mlsanity.com/paper/2306.03314/) (2023)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://mlsanity.com/paper/2305.07004/) (2023)
- [LLaMA Beyond English: An Empirical Study on Language Capability Transfer](https://mlsanity.com/paper/2401.01055/) (2024)
- [Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset](https://mlsanity.com/paper/2306.03030/) (2023)
- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [Understanding and Mitigating Language Confusion in LLMs](https://mlsanity.com/paper/2406.20052/) (2024)
- [LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback](https://mlsanity.com/paper/2406.01771/) (2024)
- [xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning](https://mlsanity.com/paper/2401.07037/) (2024)
- [Probing the Emergence of Cross-lingual Alignment during LLM Training](https://mlsanity.com/paper/2406.13229/) (2024)
- [Gender Bias in Large Language Models across Multiple Languages](https://mlsanity.com/paper/2403.00277/) (2024)
- [Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMs](https://mlsanity.com/paper/2406.09265/) (2024)
- [Bias patterns in the application of LLMs for clinical decision support: A comprehensive study](https://mlsanity.com/paper/2404.15149/) (2024)
- [Can Large Language Models Understand Context?](https://mlsanity.com/paper/2402.00858/) (2024)
- [FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domain](https://mlsanity.com/paper/2304.04280/) (2023)
- [Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustness](https://mlsanity.com/paper/2408.04585/) (2024)
- [Prompting Fairness: Integrating Causality to Debias Large Language Models](https://mlsanity.com/paper/2403.08743/) (2024)
- [Spanish and LLM Benchmarks: is MMLU Lost in Translation?](https://mlsanity.com/paper/2406.17789/) (2024)
- [Multi-FAct: Assessing Factuality of Multilingual LLMs using FActScore](https://mlsanity.com/paper/2402.18045/) (2024)
- [Healthcare Copilot: Eliciting the Power of General LLMs for Medical Consultation](https://mlsanity.com/paper/2402.13408/) (2024)
- [Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generation](https://mlsanity.com/paper/2409.11703/) (2024)
- [Enhancing Retrieval and Managing Retrieval: A Four-Module Synergy for Improved Quality and Efficiency in RAG Systems](https://mlsanity.com/paper/2407.10670/) (2024)
- [Breaking the Language Barrier: Can Direct Inference Outperform Pre-Translation in Multilingual LLM Applications?](https://mlsanity.com/paper/2403.04792/) (2024)
- [Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizer](https://mlsanity.com/paper/2305.12761/) (2023)
- [Automated Clinical Data Extraction with Knowledge Conditioned LLMs](https://mlsanity.com/paper/2406.18027/) (2024)

## Built on by (most cited first, 10 of 10)

- [MedPlan: A Two-Stage RAG-Based System for Personalized Medical Plan Generation](https://mlsanity.com/paper/2503.17900/) (2025)
- [Surfacing Subtle Stereotypes: A Multilingual, Debate-Oriented Evaluation of Modern LLMs](https://mlsanity.com/paper/2511.01187/) (2025)
- [On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI](https://mlsanity.com/paper/2508.00171/) (2025)
- [Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents](https://mlsanity.com/paper/2510.17491/) (2025)
- [Zoom In Disparities in Healthcare LLM Q&A](https://mlsanity.com/paper/2510.17476/) (2025)
- [Beyond English benchmarks: clinical llm evaluation in Brazilian Portuguese](https://mlsanity.com/paper/2606.07853/) (2026)
- [Insight-A: Attribution-aware for Multimodal Misinformation Detection](https://mlsanity.com/paper/2511.21705/) (2025)
- [Measuring Political Stance and Consistency in Large Language Models](https://mlsanity.com/paper/2601.17016/) (2026)
- [Reliable Multilingual Orthopedic Decision Support from Clinical Narratives: Language-Aware Adaptation and Verification-Guided Deferral](https://mlsanity.com/paper/2605.31512/) (2026)
- [Perspectives on Cross-Lingual Consistency in LLMs for Medical Questions](https://mlsanity.com/paper/2609.07687/) (2026)

## Abstract (arXiv)

Current ophthalmology clinical workflows are plagued by over-referrals, long waits, and complex and heterogeneous medical records. Large language models (LLMs) present a promising solution to automate various procedures such as triaging, preliminary tests like visual acuity assessment, and report summaries. However, LLMs have demonstrated significantly varied performance across different languages in natural language question-answering tasks, potentially exacerbating healthcare disparities in Low and Middle-Income Countries (LMICs). This study introduces the first multilingual ophthalmological question-answering benchmark with manually curated questions parallel across languages, allowing for direct cross-lingual comparisons. Our evaluation of 6 popular LLMs across 7 different languages reveals substantial bias across different languages, highlighting risks for clinical deployment of LLMs in LMICs. Existing debiasing methods such as Translation Chain-of-Thought or Retrieval-augmented generation (RAG) by themselves fall short of closing this performance gap, often failing to improve performance across all languages and lacking specificity for the medical domain. To address this issue, We propose CLARA (Cross-Lingual Reflective Agentic system), a novel inference time de-biasing method leveraging retrieval augmented generation and self-verification. Our approach not only improves performance across all languages but also significantly reduces the multilingual bias gap, facilitating equitable LLM application across the globe.