# CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark

arXiv: https://arxiv.org/abs/2106.08087 · HTML: https://mlsanity.com/paper/2106.08087/

## Key facts

- Cited by 101 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 49 references that are in the map.
- First posted on arXiv 15 Jun 2021.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [The Natural Language Decathlon: Multitask Learning as Question Answering](https://mlsanity.com/paper/1806.08730/) (2018)
- [CORD-19: The COVID-19 Open Research Dataset](https://mlsanity.com/paper/2004.10706/) (2020)
- [An Open Source AutoML Benchmark](https://mlsanity.com/paper/1907.00909/) (2019)
- [UER: An Open-Source Toolkit for Pre-training Models](https://mlsanity.com/paper/1909.05658/) (2019)
- [RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2010.15925/) (2020)
- [ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations](https://mlsanity.com/paper/1911.00720/) (2019)
- [Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generation](https://mlsanity.com/paper/2012.11988/) (2020)

## Built on by (most cited first, 30 of 101)

- [HuatuoGPT, towards Taming Language Model to Be a Doctor](https://mlsanity.com/paper/2305.15075/) (2023)
- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [CMB: A Comprehensive Medical Benchmark in Chinese](https://mlsanity.com/paper/2308.08833/) (2023)
- [HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs](https://mlsanity.com/paper/2311.09774/) (2023)
- [DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultation](https://mlsanity.com/paper/2308.14346/) (2023)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model](https://mlsanity.com/paper/2204.03905/) (2022)
- [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023)
- [BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPT](https://mlsanity.com/paper/2310.15896/) (2023)
- [FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2308.09975/) (2023)
- [Huatuo-26M, a Large-scale Chinese Medical QA Dataset](https://mlsanity.com/paper/2305.01526/) (2023)
- [A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasets](https://mlsanity.com/paper/2204.08997/) (2022)
- [PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain](https://mlsanity.com/paper/2310.14151/) (2023)
- [D4: a Chinese Dialogue Dataset for Depression-Diagnosis-Oriented Chat](https://mlsanity.com/paper/2205.11764/) (2022)
- [BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing](https://mlsanity.com/paper/2206.15076/) (2022)
- [Taiyi: A Bilingual Fine-Tuned Large Language Model for Diverse Biomedical Tasks](https://mlsanity.com/paper/2311.11608/) (2023)
- [LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation](https://mlsanity.com/paper/2506.04078/) (2025)
- [TC-RAG:Turing-Complete RAG's Case study on Medical LLM Systems](https://mlsanity.com/paper/2408.09199/) (2024)
- [Data-Centric Foundation Models in Computational Healthcare: A Survey](https://mlsanity.com/paper/2401.02458/) (2024)
- [Real-World Doctor Agent with Proactive Consultation through Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2505.19630/) (2025)
- [HyKGE: A Hypothesis Knowledge Graph Enhanced Framework for Accurate and Reliable Medical LLMs Responses](https://mlsanity.com/paper/2312.15883/) (2023)
- [MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Models](https://mlsanity.com/paper/2407.10990/) (2024)
- [A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry](https://mlsanity.com/paper/2404.15777/) (2024)
- [TCMBench: A Comprehensive Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine](https://mlsanity.com/paper/2406.01126/) (2024)
- [Large Language Model Benchmarks in Medical Tasks](https://mlsanity.com/paper/2410.21348/) (2024)
- [A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learning](https://mlsanity.com/paper/2504.21099/) (2025)
- [medIKAL: Integrating Knowledge Graphs as Assistants of LLMs for Enhanced Clinical Diagnosis on EMRs](https://mlsanity.com/paper/2406.14326/) (2024)
- [ContextASR-Bench: A Massive Contextual Speech Recognition Benchmark](https://mlsanity.com/paper/2507.05727/) (2025)
- [Overview of the PromptCBLUE Shared Task in CHIP2023](https://mlsanity.com/paper/2312.17522/) (2023)

## Abstract (arXiv)

Artificial Intelligence (AI), along with the recent progress in biomedical language understanding, is gradually changing medical practice. With the development of biomedical language understanding benchmarks, AI applications are widely used in the medical field. However, most benchmarks are limited to English, which makes it challenging to replicate many of the successes in English for other languages. To facilitate research in this direction, we collect real-world biomedical data and present the first Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark: a collection of natural language understanding tasks including named entity recognition, information extraction, clinical diagnosis normalization, single-sentence/sentence-pair classification, and an associated online platform for model evaluation, comparison, and analysis. To establish evaluation on these tasks, we report empirical results with the current 11 pre-trained Chinese models, and experimental results show that state-of-the-art neural models perform by far worse than the human ceiling. Our benchmark is released at urlhttps://tianchi.aliyun.com/dataset/dataDetail?dataId=95414&lang=en-us.