# MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine

arXiv: https://arxiv.org/abs/2506.01252 · HTML: https://mlsanity.com/paper/2506.01252/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 20 of 20)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs](https://mlsanity.com/paper/2412.18925/) (2024)
- [SafetyBench: Evaluating the Safety of Large Language Models](https://mlsanity.com/paper/2309.07045/) (2023)
- [CMB: A Comprehensive Medical Benchmark in Chinese](https://mlsanity.com/paper/2308.08833/) (2023)
- [Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset](https://mlsanity.com/paper/2306.03030/) (2023)
- [DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultation](https://mlsanity.com/paper/2308.14346/) (2023)
- [SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types](https://mlsanity.com/paper/2410.21965/) (2024)
- [Baichuan-M1: Pushing the Medical Capability of Large Language Models](https://mlsanity.com/paper/2502.12671/) (2025)
- [Taiyi: A Bilingual Fine-Tuned Large Language Model for Diverse Biomedical Tasks](https://mlsanity.com/paper/2311.11608/) (2023)
- [MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models](https://mlsanity.com/paper/2502.14302/) (2025)
- [BeHonest: Benchmarking Honesty in Large Language Models](https://mlsanity.com/paper/2406.13261/) (2024)
- [TCMBench: A Comprehensive Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine](https://mlsanity.com/paper/2406.01126/) (2024)
- [Qibo: A Large Language Model for Traditional Chinese Medicine](https://mlsanity.com/paper/2403.16056/) (2024)
- [TCMD: A Traditional Chinese Medicine QA Dataset for Evaluating Large Language Models](https://mlsanity.com/paper/2406.04941/) (2024)

## Built on by (most cited first, 4 of 4)

- [LingLanMiDian: Systematic Evaluation of LLMs on TCM Knowledge and Clinical Reasoning](https://mlsanity.com/paper/2602.01779/) (2026)
- [TCM-5CEval: Extended Deep Evaluation Benchmark for LLM's Comprehensive Clinical Research Competence in Traditional Chinese Medicine](https://mlsanity.com/paper/2511.13169/) (2025)
- [MMIR-TCM: Memory-Integrated Multimodal Inference and Retrieval for TCM Clinical Decision Support](https://mlsanity.com/paper/2607.01814/) (2026)
- [TSWAP: A Multilingual Retrieval-Augmented Thai Wellness Advisor](https://mlsanity.com/paper/2608.22917/) (2026)

## Abstract (arXiv)

Traditional Chinese Medicine (TCM) is a holistic medical system with millennia of accumulated clinical experience, playing a vital role in global healthcare-particularly across East Asia. However, the implicit reasoning, diverse textual forms, and lack of standardization in TCM pose major challenges for computational modeling and evaluation. Large Language Models (LLMs) have demonstrated remarkable potential in processing natural language across diverse domains, including general medicine. Yet, their systematic evaluation in the TCM domain remains underdeveloped. Existing benchmarks either focus narrowly on factual question answering or lack domain-specific tasks and clinical realism. To fill this gap, we introduce MTCMB-a Multi-Task Benchmark for Evaluating LLMs on TCM Knowledge, Reasoning, and Safety. Developed in collaboration with certified TCM experts, MTCMB comprises 12 sub-datasets spanning five major categories: knowledge QA, language understanding, diagnostic reasoning, prescription generation, and safety evaluation. The benchmark integrates real-world case records, national licensing exams, and classical texts, providing an authentic and comprehensive testbed for TCM-capable models. Preliminary results indicate that current LLMs perform well on foundational knowledge but fall short in clinical reasoning, prescription planning, and safety compliance. These findings highlight the urgent need for domain-aligned benchmarks like MTCMB to guide the development of more competent and trustworthy medical AI systems. All datasets, code, and evaluation tools are publicly available at: https://github.com/Wayyuanyuan/MTCMB.