mlsanityData to 7 Oct 2026

CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark

arXiv 2106.08087 ↗First posted 15 Jun 2021cs.CL cs.AI cs.IRcited by 99

Not on the map. Why?

Abstract (arXiv)

Artificial Intelligence (AI), along with the recent progress in biomedical language understanding, is gradually changing medical practice. With the development of biomedical language understanding benchmarks, AI applications are widely used in the medical field. However, most benchmarks are limited to English, which makes it challenging to replicate many of the successes in English for other languages. To facilitate research in this direction, we collect real-world biomedical data and present the first Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark: a collection of natural language understanding tasks including named entity recognition, information extraction, clinical diagnosis normalization, single-sentence/sentence-pair classification, and an associated online platform for model evaluation, comparison, and analysis. To establish evaluation on these tasks, we report empirical results with the current 11 pre-trained Chinese models, and experimental results show that state-of-the-art neural models perform by far worse than the human ceiling. Our benchmark is released at urlhttps://tianchi.aliyun.com/dataset/dataDetail?dataId=95414&lang=en-us.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Builds on 22 of its 49 references that are in the map. First posted on arXiv 15 Jun 2021.

Builds on

22 of 49 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  9. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  10. 2018Datasheets for Datasetscited by 1,371
  11. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  12. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
10 more
  1. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  2. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  3. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 262
  4. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  5. 2020CORD-19: The COVID-19 Open Research Datasetcited by 179
  6. 2019An Open Source AutoML Benchmarkcited by 85
  7. 2019UER: An Open-Source Toolkit for Pre-training Modelscited by 61
  8. 2020RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmarkcited by 53
  9. 2019ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representationscited by 35
  10. 2020Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generationcited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 11; 2023: 26; 2024: 32; 2025: 17; 2026: 11.

99 papers.

Sort:

By area:

  1. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 211
  2. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  3. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  4. 2023CMB: A Comprehensive Medical Benchmark in Chinesecited by 113
  5. 2023HuatuoGPT-II, One-stage Training for Medical Adaption of LLMscited by 91
  6. 2023DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultationcited by 89
  7. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  8. 2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
  9. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  10. 2023BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPTcited by 64
  11. 2023FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Modelscited by 52
  12. 2023Huatuo-26M, a Large-scale Chinese Medical QA Datasetcited by 46
38 more
  1. 2022A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasetscited by 45
  2. 2023PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domaincited by 40
  3. 2022D4: a Chinese Dialogue Dataset for Depression-Diagnosis-Oriented Chatcited by 37
  4. 2022BigBIO: A Framework for Data-Centric Biomedical Natural Language Processingcited by 36
  5. 2023Taiyi: A Bilingual Fine-Tuned Large Language Model for Diverse Biomedical Taskscited by 36
  6. 2025LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validationcited by 29
  7. 2024TC-RAG:Turing-Complete RAG's Case study on Medical LLM Systemscited by 28
  8. 2024Data-Centric Foundation Models in Computational Healthcare: A Surveycited by 27
  9. 2025Real-World Doctor Agent with Proactive Consultation through Multi-Agent Reinforcement Learningcited by 26
  10. 2023HyKGE: A Hypothesis Knowledge Graph Enhanced Framework for Accurate and Reliable Medical LLMs Responsescited by 25
  11. 2024MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Modelscited by 21
  12. 2024A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industrycited by 17
  13. 2024TCMBench: A Comprehensive Benchmark for Evaluating Large Language Models in Traditional Chinese Medicinecited by 16
  14. 2024medIKAL: Integrating Knowledge Graphs as Assistants of LLMs for Enhanced Clinical Diagnosis on EMRscited by 16
  15. 2024Large Language Model Benchmarks in Medical Taskscited by 16
  16. 2025A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learningcited by 15
  17. 2023Overview of the PromptCBLUE Shared Task in CHIP2023cited by 14
  18. 2025ContextASR-Bench: A Massive Contextual Speech Recognition Benchmarkcited by 14
  19. 2022This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polishcited by 13
  20. 2022Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learningcited by 11
  21. 2024DrBenchmark: A Large Language Understanding Evaluation Benchmark for French Biomedical Domaincited by 11
  22. 2022RuMedBench: A Russian Medical Language Understanding Benchmarkcited by 10
  23. 2023WYWEB: A NLP Evaluation Benchmark For Classical Chinesecited by 10
  24. 2024Role Prompting Guided Domain Adaptation with General Capability Preserve for Large Language Modelscited by 10
  25. 2023Intent Identification and Entity Extraction for Healthcare Queries in Indic Languagescited by 9
  26. 2024A Fast, Performant, Secure Distributed Training Framework For Large Language Modelcited by 8
  27. 2025MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agentscited by 8
  28. 2024MING-MOE: Enhancing Medical Multi-Task Learning in Large Language Models with Sparse Mixture of Low-Rank Adapter Expertscited by 7
  29. 2024Text2MDT: Extracting Medical Decision Trees from Medical Textscited by 6
  30. 2024Intent Detection and Entity Extraction from BioMedical Literaturecited by 6
  31. 2025AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Fieldcited by 6
  32. 2021Building Chinese Biomedical Language Models via Multi-Level Text Discriminationcited by 5
  33. 2022Towards Realistic Low-resource Relation Extraction: A Benchmark with Empirical Baseline Studycited by 5
  34. 2023GeoGLUE: A GeoGraphic Language Understanding Evaluation Benchmarkcited by 5
  35. 2024Medical Dialogue: A Survey of Categories, Methods, Evaluation and Challengescited by 5
  36. 2024MultifacetEval: Multifaceted Evaluation to Probe LLMs in Mastering Medical Knowledgecited by 5
  37. 2024TCMD: A Traditional Chinese Medicine QA Dataset for Evaluating Large Language Modelscited by 5
  38. 2024Beyond Boundaries: Learning a Universal Entity Taxonomy across Datasets and Languages for Open Named Entity Recognitioncited by 5

50 most cited of 99 All 99 (JSON)