# MedFact: Benchmarking the Fact-Checking Capabilities of Large Language Models on Chinese Medical Texts

arXiv: https://arxiv.org/abs/2509.12440 · HTML: https://mlsanity.com/paper/2509.12440/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2025.

## Built on by (most cited first, 1 of 1)

- [MedSNIP: Building and Benchmarking Snippet-Level Granularity for Medical Fact Verification](https://mlsanity.com/paper/2609.12884/) (2026)

## Abstract (arXiv)

Deploying Large Language Models (LLMs) in medical applications requires fact-checking capabilities to ensure patient safety and regulatory compliance. We introduce MedFact, a challenging Chinese medical fact-checking benchmark with 2,116 expert-annotated instances from diverse real-world texts, spanning 13 specialties, 8 error types, 4 writing styles, and 5 difficulty levels. Construction uses a hybrid AI-human framework where iterative expert feedback refines AI-driven, multi-criteria filtering to ensure high quality and difficulty. We evaluate 20 leading LLMs on veracity classification and error localization, and results show models often determine if text contains errors but struggle to localize them precisely, with top performers falling short of human performance. Our analysis reveals the "over-criticism" phenomenon, a tendency for models to misidentify correct information as erroneous, which can be exacerbated by advanced reasoning techniques such as multi-agent collaboration and inference-time scaling. MedFact highlights the challenges of deploying medical LLMs and provides resources to develop factually reliable medical AI systems.