mlsanityData to 7 Oct 2026

HealMed: Multilingual Evaluation of Large Language Models in Medicine

arXiv 2608.19981 ↗First posted 20 Aug 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We present HealMed, an expert-reviewed benchmark for multilingual evaluation of large language models in medicine. HealMed contains 1,000 examples in each of nine languages, drawn from nine datasets and covering three task formats: MCQA, NLI and open-ended QA. The benchmark was developed over two years by 23 physicians and medical experts based across nine countries and regions. Each translation was evaluated and revised by two experts fluent in English and the corresponding target language. On HealMed, performance declined most in low-resource languages, although the size of the gap varied markedly across languages and models. The strongest proprietary models were the most stable across languages, whereas many open-source and medically specialized models showed larger and less consistent gaps. Medical specialization alone did not ensure multilingual robustness. Furthermore, expert revision could either raise or lower measured performance, indicating that translation quality materially affects cross-language evaluation results.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 32 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

20 of 32 references.

References · 20

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  4. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  5. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  6. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  7. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  8. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 181
  9. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 149
  10. 2019HEAD-QA: A Healthcare Dataset for Complex Reasoningcited by 100
  11. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 96
  12. 2020Translation Artifacts in Cross-lingual Transfer Learningcited by 95
8 more
  1. 2024Towards Building Multilingual Language Model for Medicinecited by 91
  2. 2023Benchmarking large language models for biomedical natural language processing applications and recommendationscited by 78
  3. 2024Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B Peoplecited by 38
  4. 2025The Bitter Lesson Learned from 2,000+ Multilingual Benchmarkscited by 31
  5. 2026HealthBench Professional: Evaluating Large Language Models on Real Clinician Chatscited by 22
  6. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  7. 2022BioNLI: Generating a Biomedical NLI Dataset Using Lexico-semantic Constraints for Adversarial Examplescited by 15
  8. 2026Med-CoReasoner: Reducing Language Disparities in Medical Reasoning via Language-Informed Co-Reasoningcited by 2

Built on by

Not cited yet.