# LiveMedBench: A Contamination-Free Medical Benchmark for LLMs with Automated Rubric Evaluation

arXiv: https://arxiv.org/abs/2602.10367 · HTML: https://mlsanity.com/paper/2602.10367/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Builds on (most cited first, 19 of 19)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://mlsanity.com/paper/2309.15217/) (2023)
- [Data Contamination: From Memorization to Exploitation](https://mlsanity.com/paper/2203.08242/) (2022)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [CMB: A Comprehensive Medical Benchmark in Chinese](https://mlsanity.com/paper/2308.08833/) (2023)
- [MedRAG: Enhancing Retrieval-augmented Generation with Knowledge Graph-Elicited Reasoning for Healthcare Copilot](https://mlsanity.com/paper/2502.04413/) (2025)
- [Baichuan-M2: Scaling Medical Capability with Large Verifier System](https://mlsanity.com/paper/2509.02208/) (2025)
- [MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models](https://mlsanity.com/paper/2312.12806/) (2023)
- [CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios](https://mlsanity.com/paper/2410.03502/) (2024)
- [Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluation](https://mlsanity.com/paper/2510.09275/) (2025)
- [The MedPerturb Dataset: What Non-Content Perturbations Reveal About Human and Clinical LLM Decision Making](https://mlsanity.com/paper/2506.17163/) (2025)

## Built on by (most cited first, 12 of 12)

- [Experience Makes Skillful: Enabling Generalizable Medical Agent Reasoning via Self-Evolving Skill Memory](https://mlsanity.com/paper/2606.09365/) (2026)
- [MedCTA: A Benchmark for Clinical Tool Agents](https://mlsanity.com/paper/2606.11702/) (2026)
- [CT Open: An Open-Access, Uncontaminated, Live Platform for the Open Challenge of Clinical Trial Outcome Prediction](https://mlsanity.com/paper/2604.16742/) (2026)
- [MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters](https://mlsanity.com/paper/2607.08257/) (2026)
- [RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty](https://mlsanity.com/paper/2602.12424/) (2026)
- [When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries](https://mlsanity.com/paper/2606.28332/) (2026)
- [MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation](https://mlsanity.com/paper/2607.09142/) (2026)
- [APTER: Adaptive Post-Training with Expert-Grounded Rubrics](https://mlsanity.com/paper/2608.14212/) (2026)
- [Dense Clinical Contrasts Enhance Medical Knowledge Updating in Large Language Models](https://mlsanity.com/paper/2608.30405/) (2026)
- [A primer on evaluation methods for large language models in healthcare](https://mlsanity.com/paper/2609.14819/) (2026)
- [Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks](https://mlsanity.com/paper/2609.16023/) (2026)
- [Self-Evolving Agent Harness](https://mlsanity.com/paper/2610.18813/) (2026)

## Abstract (arXiv)

The deployment of Large Language Models (LLMs) in high-stakes clinical settings demands rigorous and reliable evaluation. However, existing medical benchmarks remain static, suffering from two critical limitations: (1) data contamination, where test sets inadvertently leak into training corpora, leading to inflated performance estimates; and (2) temporal misalignment, failing to capture the rapid evolution of medical knowledge. Furthermore, current evaluation metrics for open-ended clinical reasoning often rely on either shallow lexical overlap (e.g., ROUGE) or subjective LLM-as-a-Judge scoring, both inadequate for verifying clinical correctness. To bridge these gaps, we introduce LiveMedBench, a continuously updated, contamination-free, and rubric-based benchmark that weekly harvests real-world clinical cases from online medical communities, ensuring strict temporal separation from model training data. We propose a Multi-Agent Clinical Curation Framework that filters raw data noise and validates clinical integrity against evidence-based medical principles. For evaluation, we develop an Automated Rubric-based Evaluation Framework that decomposes physician responses into granular, case-specific criteria, achieving substantially stronger alignment with expert physicians than LLM-as-a-Judge. To date, LiveMedBench comprises 2,756 real-world cases spanning 38 medical specialties and multiple languages, paired with 16,702 unique evaluation criteria. Extensive evaluation of 38 LLMs reveals that even the best-performing model achieves only 39.2%, and 84% of models exhibit performance degradation on post-cutoff cases, confirming pervasive data contamination risks. Error analysis further identifies contextual application-not factual knowledge-as the dominant bottleneck, with 35-48% of failures stemming from the inability to tailor medical knowledge to patient-specific constraints.