mlsanityData to 7 Oct 2026

CRAB: A Benchmark for Evaluating Curation of Retrieval-Augmented LLMs in Biomedicine

arXiv 2504.12342 ↗First posted 15 Apr 2025cs.CLcited by 0

On the map: Retrieval-augmented generation → Medical and biomedical RAG

Abstract (arXiv)

Recent development in Retrieval-Augmented Large Language Models (LLMs) have shown great promise in biomedical applications. How ever, a critical gap persists in reliably evaluating their curation ability the process by which models select and integrate relevant references while filtering out noise. To address this, we introduce the benchmark for Curation of Retrieval-Augmented LLMs in Biomedicine (CRAB), the first multilingual benchmark tailored for evaluating the biomedical curation of retrieval-augmented LLMs, available in English, French, German and Chinese. By incorporating a novel citation-based evaluation metric, CRAB quantifies the curation performance of retrieval-augmented LLMs in biomedicine. Experimental results reveal significant discrepancies in the curation performance of mainstream LLMs, underscoring the urgent need to improve it in the domain of biomedicine. Our dataset is available at https://huggingface.co/datasets/zhm0/CRAB.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. Builds on 39 of its 46 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

39 of 46 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answering14 Nov
  2. 2024MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering8 Apr
  3. 2024Benchmarking Retrieval-Augmented Generation for Medicine20 Feb
  4. 2023MKRAG: Medical Knowledge Retrieval Augmented Generation for Medical Question Answering27 Sep
  5. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicine1 Mar

Other references · 34

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  9. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  10. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
22 more
  1. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  2. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  3. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  4. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  5. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  6. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  7. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  8. 2022Teaching language models to support answers with verified quotescited by 290
  9. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  10. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  11. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  12. 2022Large Language Models with Controllable Working Memorycited by 169
  13. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  14. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  15. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  16. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 143
  17. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 115
  18. 2020Factual Error Correction for Abstractive Summarization Modelscited by 108
  19. 2023Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Datasetcited by 90
  20. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Trainingcited by 88
  21. 2023Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Taskscited by 59
  22. 2024RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generationcited by 59

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024BiomedRAG: A Retrieval Augmented Large Language Model for Biomedicine
  2. 2024A Review on Scientific Knowledge Extraction using Large Language Models in Biomedical Sciences
  3. 2024Benchmarking Retrieval-Augmented Large Language Models in Biomedical NLP: Application, Robustness, and Self-Awareness
  4. 2025Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights
  5. 2026MMed-Bench-IR: A Heterogeneous Benchmark for Multilingual Medical Information Retrieval
  6. 2026Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study
  7. 2026MRAG: Benchmarking Retrieval-Augmented Generation for Bio-medicine
  8. 2024Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data