# Clinical Reading Comprehension: A Thorough Analysis of the emrQA Dataset

arXiv: https://arxiv.org/abs/2005.00574 · HTML: https://mlsanity.com/paper/2005.00574/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 May 2020.

## Built on by (most cited first, 23 of 23)

- [GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records](https://mlsanity.com/paper/2203.03540/) (2022)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences](https://mlsanity.com/paper/2201.11838/) (2022)
- [MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records](https://mlsanity.com/paper/2308.14089/) (2023)
- [Biomedical Question Answering: A Survey of Approaches and Challenges](https://mlsanity.com/paper/2102.05281/) (2021)
- [A Comparative Study of Pretrained Language Models for Long Clinical Text](https://mlsanity.com/paper/2301.11847/) (2023)
- [EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries](https://mlsanity.com/paper/2402.16040/) (2024)
- [Large Language Models are Clinical Reasoners: Reasoning-Aware Diagnosis Framework with Prompt-Generated Rationales](https://mlsanity.com/paper/2312.07399/) (2023)
- [The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRs](https://mlsanity.com/paper/2303.12961/) (2023)
- [KALA: Knowledge-Augmented Language Model Adaptation](https://mlsanity.com/paper/2204.10555/) (2022)
- [PaniniQA: Enhancing Patient Education Through Interactive Question Answering](https://mlsanity.com/paper/2308.03253/) (2023)
- [Learning to Ask Like a Physician](https://mlsanity.com/paper/2206.02696/) (2022)
- [DR.BENCH: Diagnostic Reasoning Benchmark for Clinical Natural Language Processing](https://mlsanity.com/paper/2209.14901/) (2022)
- [A Scoping Review of Publicly Available Language Tasks in Clinical Natural Language Processing](https://mlsanity.com/paper/2112.05780/) (2021)
- [ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination](https://mlsanity.com/paper/2305.12945/) (2023)
- [Question Answering for Electronic Health Records: A Scoping Review of datasets and models](https://mlsanity.com/paper/2310.08759/) (2023)
- [TCM-SD: A Benchmark for Probing Syndrome Differentiation via Natural Language Processing](https://mlsanity.com/paper/2203.10839/) (2022)
- [Development of an Extractive Clinical Question Answering Dataset with Multi-Answer and Multi-Focus Questions](https://mlsanity.com/paper/2201.02517/) (2022)
- [KU-DMIS at EHRSQL 2024:Generating SQL query via question templatization in EHR](https://mlsanity.com/paper/2406.00014/) (2024)
- [Towards More Robust Natural Language Understanding](https://mlsanity.com/paper/2112.02992/) (2021)
- [Toward a Neural Semantic Parsing System for EHR Question Answering](https://mlsanity.com/paper/2211.04569/) (2022)
- [A Unified Knowledge Graph Augmentation Service for Boosting Domain-specific NLP Tasks](https://mlsanity.com/paper/2212.05251/) (2022)
- [CLIFT: Analysing Natural Distribution Shift on Question Answering Models in Clinical Domain](https://mlsanity.com/paper/2310.13146/) (2023)

## Abstract (arXiv)

Machine reading comprehension has made great progress in recent years owing to large-scale annotated datasets. In the clinical domain, however, creating such datasets is quite difficult due to the domain expertise required for annotation. Recently, Pampari et al. (EMNLP'18) tackled this issue by using expert-annotated question templates and existing i2b2 annotations to create emrQA, the first large-scale dataset for question answering (QA) based on clinical notes. In this paper, we provide an in-depth analysis of this dataset and the clinical reading comprehension (CliniRC) task. From our qualitative analysis, we find that (i) emrQA answers are often incomplete, and (ii) emrQA questions are often answerable without using domain knowledge. From our quantitative experiments, surprising results include that (iii) using a small sampled subset (5%-20%), we can obtain roughly equal performance compared to the model trained on the entire dataset, (iv) this performance is close to human expert's performance, and (v) BERT models do not beat the best performing base model. Following our analysis of the emrQA, we further explore two desired aspects of CliniRC systems: the ability to utilize clinical domain knowledge and to generalize to unseen questions and contexts. We argue that both should be considered when creating future datasets.