# ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination

arXiv: https://arxiv.org/abs/2305.12945 · HTML: https://mlsanity.com/paper/2305.12945/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 49 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [Evaluating the Performance of Large Language Models on GAOKAO Benchmark](https://mlsanity.com/paper/2305.12474/) (2023)
- [The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning](https://mlsanity.com/paper/2205.03401/) (2022)
- [emrQA: A Large Corpus for Question Answering on Electronic Medical Records](https://mlsanity.com/paper/1809.00732/) (2018)
- [Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing](https://mlsanity.com/paper/2102.12060/) (2021)
- [Reframing Human-AI Collaboration for Generating Free-Text Explanations](https://mlsanity.com/paper/2112.08674/) (2021)
- [Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases](https://mlsanity.com/paper/2303.14742/) (2023)
- [Huatuo-26M, a Large-scale Chinese Medical QA Dataset](https://mlsanity.com/paper/2305.01526/) (2023)
- [Causal-Discovery Performance of ChatGPT in the context of Neuropathic Pain Diagnosis](https://mlsanity.com/paper/2301.13819/) (2023)
- [Medical Exam Question Answering with Large-scale Reading Comprehension](https://mlsanity.com/paper/1802.10279/) (2018)
- [Putting ChatGPT's Medical Advice to the (Turing) Test](https://mlsanity.com/paper/2301.10035/) (2023)
- [Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset](https://mlsanity.com/paper/2005.00574/) (2020)

## Built on by (most cited first, 10 of 10)

- [CMB: A Comprehensive Medical Benchmark in Chinese](https://mlsanity.com/paper/2308.08833/) (2023)
- [Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions](https://mlsanity.com/paper/2402.18060/) (2024)
- [Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey](https://mlsanity.com/paper/2311.07914/) (2023)
- [MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Models](https://mlsanity.com/paper/2308.09729/) (2023)
- [Automating Expert-Level Medical Reasoning Evaluation of Large Language Models](https://mlsanity.com/paper/2507.07988/) (2025)
- [An Enhanced Prompt-Based LLM Reasoning Scheme via Knowledge Graph-Integrated Collaboration](https://mlsanity.com/paper/2402.04978/) (2024)
- [DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models](https://mlsanity.com/paper/2408.01933/) (2024)
- [Deciphering Diagnoses: How Large Language Models Explanations Influence Clinical Decision Making](https://mlsanity.com/paper/2310.01708/) (2023)
- [Towards Self-cognitive Exploration: Metacognitive Knowledge Graph Retrieval Augmented Generation](https://mlsanity.com/paper/2508.09460/) (2025)
- [A Pilot Empirical Study on When and How to Use Knowledge Graphs as Retrieval Augmented Generation](https://mlsanity.com/paper/2502.20854/) (2025)

## Abstract (arXiv)

As ChatGPT and GPT-4 spearhead the development of Large Language Models (LLMs), more researchers are investigating their performance across various tasks. But more research needs to be done on the interpretability capabilities of LLMs, that is, the ability to generate reasons after an answer has been given. Existing explanation datasets are mostly English-language general knowledge questions, which leads to insufficient thematic and linguistic diversity. To address the language bias and lack of medical resources in generating rationales QA datasets, we present ExplainCPE (over 7k instances), a challenging medical benchmark in Simplified Chinese. We analyzed the errors of ChatGPT and GPT-4, pointing out the limitations of current LLMs in understanding text and computational reasoning. During the experiment, we also found that different LLMs have different preferences for in-context learning. ExplainCPE presents a significant challenge, but its potential for further investigation is promising, and it can be used to evaluate the ability of a model to generate explanations. AI safety and trustworthiness need more attention, and this work makes the first step to explore the medical interpretability of LLMs.The dataset is available at https://github.com/HITsz-TMG/ExplainCPE.