# TCMBench: A Comprehensive Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine

arXiv: https://arxiv.org/abs/2406.01126 · HTML: https://mlsanity.com/paper/2406.01126/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 22 references that are in the map.
- First posted on arXiv 3 Jun 2024.

## Builds on (most cited first, 9 of 9)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge](https://mlsanity.com/paper/2304.06975/) (2023)
- [CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2106.08087/) (2021)
- [PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain](https://mlsanity.com/paper/2310.14151/) (2023)
- [MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models](https://mlsanity.com/paper/2312.12806/) (2023)

## Built on by (most cited first, 17 of 17)

- [TCM-Ladder: A Benchmark for Multimodal Question Answering on Traditional Chinese Medicine](https://mlsanity.com/paper/2505.24063/) (2025)
- [TCM-3CEval: A Triaxial Benchmark for Assessing Responses from Large Language Models in Traditional Chinese Medicine](https://mlsanity.com/paper/2503.07041/) (2025)
- [MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicine](https://mlsanity.com/paper/2506.01252/) (2025)
- [Traditional Chinese Medicine Case Analysis System for High-Level Semantic Abstraction: Optimized with Prompt and RAG](https://mlsanity.com/paper/2411.15491/) (2024)
- [OphthBench: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Ophthalmology](https://mlsanity.com/paper/2502.01243/) (2025)
- [Improving TCM Question Answering through Tree-Organized Self-Reflective Retrieval with LLMs](https://mlsanity.com/paper/2502.09156/) (2025)
- [Reviewing Clinical Knowledge in Medical Large Language Models: Training and Beyond](https://mlsanity.com/paper/2502.20988/) (2025)
- [TeleEval-OS: Performance evaluations of large language models for operations scheduling](https://mlsanity.com/paper/2506.11017/) (2025)
- [Overview of CHIP 2025 Shared Task 2: Discharge Medication Recommendation for Metabolic Diseases Based on Chinese Electronic Health Records](https://mlsanity.com/paper/2511.06230/) (2025)
- [LingLanMiDian: Systematic Evaluation of LLMs on TCM Knowledge and Clinical Reasoning](https://mlsanity.com/paper/2602.01779/) (2026)
- [FAM-Bench: A Multimodal Benchmark for Condition-Aware Food-as-Medicine Reasoning](https://mlsanity.com/paper/2605.31410/) (2026)
- [TCM-5CEval: Extended Deep Evaluation Benchmark for LLM's Comprehensive Clinical Research Competence in Traditional Chinese Medicine](https://mlsanity.com/paper/2511.13169/) (2025)
- [MMIR-TCM: Memory-Integrated Multimodal Inference and Retrieval for TCM Clinical Decision Support](https://mlsanity.com/paper/2607.01814/) (2026)
- [TreeProbe : A Tibetan Medicine Benchmark for Cultural Bias in LLMs](https://mlsanity.com/paper/2608.00640/) (2026)
- [TCMaster: Confidence-Aware Querying and Workload-Guided Physical Design for Multi-Source Traditional Chinese Medicine Knowledge Graphs](https://mlsanity.com/paper/2609.25712/) (2026)
- [Syndrome, Synergy, and Safety: Structured Reasoning and Knowledge-Driven Alignment for TCM Prescription Generation](https://mlsanity.com/paper/2609.25755/) (2026)
- [TCMQA: A 38K-Question Traditional Chinese Medicine Benchmark with a Licensed-Practitioner Reference](https://mlsanity.com/paper/2609.33014/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have performed remarkably well in various natural language processing tasks by benchmarking, including in the Western medical domain. However, the professional evaluation benchmarks for LLMs have yet to be covered in the traditional Chinese medicine(TCM) domain, which has a profound history and vast influence. To address this research gap, we introduce TCM-Bench, an comprehensive benchmark for evaluating LLM performance in TCM. It comprises the TCM-ED dataset, consisting of 5,473 questions sourced from the TCM Licensing Exam (TCMLE), including 1,300 questions with authoritative analysis. It covers the core components of TCMLE, including TCM basis and clinical practice. To evaluate LLMs beyond accuracy of question answering, we propose TCMScore, a metric tailored for evaluating the quality of answers generated by LLMs for TCM related questions. It comprehensively considers the consistency of TCM semantics and knowledge. After conducting comprehensive experimental analyses from diverse perspectives, we can obtain the following findings: (1) The unsatisfactory performance of LLMs on this benchmark underscores their significant room for improvement in TCM. (2) Introducing domain knowledge can enhance LLMs' performance. However, for in-domain models like ZhongJing-TCM, the quality of generated analysis text has decreased, and we hypothesize that their fine-tuning process affects the basic LLM capabilities. (3) Traditional metrics for text generation quality like Rouge and BertScore are susceptible to text length and surface semantic ambiguity, while domain-specific metrics such as TCMScore can further supplement and explain their evaluation results. These findings highlight the capabilities and limitations of LLMs in the TCM and aim to provide a more profound assistance to medical research.