mlsanityData to 7 Oct 2026

MedMeta: A Benchmark for LLMs in Synthesizing Meta-Analysis Conclusion from Medical Studies

arXiv 2605.09661 ↗First posted 10 May 2026cs.CL cs.AIcited by 0

On the map: Retrieval-augmented generation → Medical and biomedical RAG

Abstract (arXiv)

Large language models (LLMs) have saturated standard medical benchmarks that test factual recall, yet their ability to perform higher-order reasoning, such as synthesizing evidence from multiple sources, remains critically under-explored. To address this gap, we introduce MedMeta, the first benchmark designed to evaluate an LLM's ability to generate conclusions from medical meta-analyses using only the abstracts of cited studies. MedMeta comprises 81 meta-analyses from PubMed (2018--2025) and evaluates models using two distinct workflows: a Retrieval-Augmented Generation (Golden-RAG) setting with ground-truth abstracts, and a Parametric-only approach relying on internal knowledge. Our evaluation framework is validated by a well-structured analysis showing our LLM-as-a-judge protocol strongly aligns with human expert ratings, as evidenced by high Pearson's r correlation (0.81) and Bland-Altman analysis revealing negligible systematic bias, establishing it as a reliable proxy for scalable evaluation. Our findings underscore the critical importance of information grounding: the Golden-RAG workflow consistently and significantly outperforms the Parametric-only approach across models. In contrast, the benefits of domain-specific fine-tuning are marginal and largely neutralized when external material is provided. Furthermore, stress tests show that all models, regardless of architecture, fail to identify and reject negated evidence, highlighting a critical vulnerability in current RAG systems. Notably, even under ideal RAG conditions, current LLMs achieve only slightly above-average performance (~2.7/5.0). MedMeta provides a challenging new benchmark for evidence synthesis and demonstrates that for clinical applications, developing robust RAG systems is a more promising direction than model specialization alone.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. Builds on 33 of its 43 references that are in the map. First posted on arXiv 10 May 2026.

Builds on

33 of 43 references.

References · 33

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  8. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  9. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  11. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  12. 2022Quantifying Memorization Across Neural Language Modelscited by 755
21 more
  1. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  2. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  3. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  4. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  5. 2023Towards Generalist Biomedical AIcited by 286
  6. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  7. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  8. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 212
  9. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 180
  10. 2023Med-HALT: Medical Domain Hallucination Test for Large Language Modelscited by 156
  11. 2024Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questionscited by 106
  12. 2024AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulatorcited by 99
  13. 2024Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Trainingcited by 88
  14. 2020Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databasescited by 75
  15. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  16. 2024MedExQA: Medical Question Answering Benchmark with Multiple Explanationscited by 58
  17. 2025MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agentscited by 56
  18. 2023Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noisecited by 52
  19. 2023RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledgecited by 51
  20. 2024EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summariescited by 33
  21. 2024Reinforcement Retrieval Leveraging Fine-grained Feedback for Fact Checking News Claims with Black-Box LLMcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Can Large Language Models Match the Conclusions of Systematic Reviews?
  2. 2025META-RAG: Meta-Analysis-Inspired Evidence-Re-Ranking Method for Retrieval-Augmented Generation in Evidence-Based Medicine
  3. 2025Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights
  4. 2024Empowering Meta-Analysis: Leveraging Large Language Models for Scientific Synthesis
  5. 2025Evaluating Large Language Models for Evidence-Based Clinical Question Answering
  6. 2024A Review on Scientific Knowledge Extraction using Large Language Models in Biomedical Sciences
  7. 2026CLEAR: Cross-Source Evidence Adjudication for Large Language Models in Medicine
  8. 2026Towards Reliable Medical LLMs: Benchmarking and Enhancing Confidence Estimation of Large Language Models in Medical Consultation