MMed-Bench-IR: A Heterogeneous Benchmark for Multilingual Medical Information Retrieval
On the map: Retrieval-augmented generation → Medical and biomedical RAG
Abstract (arXiv)
Retrieval-augmented generation (RAG) in clinical settings increasingly requires multilingual retrieval against predominantly English evidence corpora. Multilingual medical retrieval demands three capabilities: cross-lingual alignment, concept discrimination, and evidence retrieval. However, existing benchmarks evaluate these only in isolation, leaving the interaction between biomedical expertise and multilingual coverage unmeasured. We introduce MMed-Bench-IR, a benchmark designed to disentangle these axes across 6 languages and three structurally heterogeneous tasks: (1) cross-lingual medical QA retrieval with 6,127 queries grounded in the Unified Medical Language System (UMLS), (2) concept discrimination over 4,975 confusion sets at three difficulty tiers, and (3) multilingual evidence retrieval for RAG with 2,040 quality-assured queries. The three tasks share zero concept and query overlap by design, ensuring that aggregate scores reflect genuine capability breadth. Evaluation of ten systems across six paradigm families reveals severe cross-lingual failure: biomedical encoders that score 0.818 nDCG@10 in English drop to 0.056 in Japanese, a gap that English-only benchmarks cannot detect.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. Builds on 14 of its 27 references that are in the map. First posted on arXiv 23 Jun 2026.
Builds on
14 of 27 references.
References · 14
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
- 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
- 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
- 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
- 2020Self-Alignment Pretraining for Biomedical Entity Representationscited by 237
- 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 185
- 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
- 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 119
- 2024Towards Building Multilingual Language Model for Medicinecited by 92
- 2020CODER: Knowledge infused cross-lingual medical term embedding for term normalizationcited by 55
- 2023BioLORD-2023: Semantic Textual Representations Fusing LLM and Clinical Knowledge Graph Insightscited by 7
2 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Retrieval-Augmented Generation in Medicine: A Scoping Review of Technical Implementations, Clinical Applications, and Ethical Considerations
- 2026Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study
- 2025MKG-Rank: Enhancing Large Language Models with Knowledge Graph for Multilingual Medical Question Answering
- 2025Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights
- 2025Multilingual Medical Reasoning for Question Answering with Large Language Models
- 2026MRAG: Benchmarking Retrieval-Augmented Generation for Bio-medicine
- 2025CRAB: A Benchmark for Evaluating Curation of Retrieval-Augmented LLMs in Biomedicine
- 2023MKRAG: Medical Knowledge Retrieval Augmented Generation for Medical Question Answering