# BIOMRC: A Dataset for Biomedical Machine Reading Comprehension

arXiv: https://arxiv.org/abs/2005.06376 · HTML: https://mlsanity.com/paper/2005.06376/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2020.

## Built on by (most cited first, 30 of 35)

- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [Document Understanding Dataset and Evaluation (DUDE)](https://mlsanity.com/paper/2305.08455/) (2023)
- [SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers](https://mlsanity.com/paper/2407.09413/) (2024)
- [A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks](https://mlsanity.com/paper/2407.12994/) (2024)
- [Biomedical Question Answering: A Survey of Approaches and Challenges](https://mlsanity.com/paper/2102.05281/) (2021)
- [BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing](https://mlsanity.com/paper/2206.15076/) (2022)
- [Exploring Universal Intrinsic Task Subspace via Prompt Tuning](https://mlsanity.com/paper/2110.07867/) (2021)
- [SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers](https://mlsanity.com/paper/2411.05338/) (2024)
- [Human Still Wins over LLM: An Empirical Study of Active Learning on Domain-Specific Annotation Tasks](https://mlsanity.com/paper/2311.09825/) (2023)
- [Sequence tagging for biomedical extractive question answering](https://mlsanity.com/paper/2104.07535/) (2021)
- [Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Experts](https://mlsanity.com/paper/2205.12701/) (2022)
- [Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?](https://mlsanity.com/paper/2302.08143/) (2023)
- [SCITAT: A Question Answering Benchmark for Scientific Tables and Text Covering Diverse Reasoning Types](https://mlsanity.com/paper/2412.11757/) (2024)
- [Can LLMs Augment Low-Resource Reading Comprehension Datasets? Opportunities and Challenges](https://mlsanity.com/paper/2309.12426/) (2023)
- [MoDEM: Mixture of Domain Expert Models](https://mlsanity.com/paper/2410.07490/) (2024)
- [NuclearQA: A Human-Made Benchmark for Language Models for the Nuclear Domain](https://mlsanity.com/paper/2310.10920/) (2023)
- [Few-shot Adaptation Works with UnpredicTable Data](https://mlsanity.com/paper/2208.01009/) (2022)
- [Data Augmentation for Biomedical Factoid Question Answering](https://mlsanity.com/paper/2204.04711/) (2022)
- [Comparing Template-based and Template-free Language Model Probing](https://mlsanity.com/paper/2402.00123/) (2024)
- [Contextual embedding and model weighting by fusing domain knowledge on Biomedical Question Answering](https://mlsanity.com/paper/2206.12866/) (2022)
- [WeQA: A Benchmark for Retrieval Augmented Generation in Wind Energy Domain](https://mlsanity.com/paper/2408.11800/) (2024)
- [M-QALM: A Benchmark to Assess Clinical Reading Comprehension and Knowledge Recall in Large Language Models via Question Answering](https://mlsanity.com/paper/2406.03699/) (2024)
- [Optimized Biomedical Question-Answering Services with LLM and Multi-BERT Integration](https://mlsanity.com/paper/2410.12856/) (2024)
- [From Queries to Criteria: Understanding How Astronomers Evaluate LLMs](https://mlsanity.com/paper/2507.15715/) (2025)
- [ArgSciChat: A Dataset for Argumentative Dialogues on Scientific Papers](https://mlsanity.com/paper/2202.06690/) (2022)
- [KnowDA: All-in-One Knowledge Mixture Model for Data Augmentation in Low-Resource NLP](https://mlsanity.com/paper/2206.10265/) (2022)
- [Can GPT Redefine Medical Understanding? Evaluating GPT on Biomedical Machine Reading Comprehension](https://mlsanity.com/paper/2405.18682/) (2024)
- [Recent Advances in Automated Question Answering In Biomedical Domain](https://mlsanity.com/paper/2111.05937/) (2021)

## Abstract (arXiv)

We introduce BIOMRC, a large-scale cloze-style biomedical MRC dataset. Care was taken to reduce noise, compared to the previous BIOREAD dataset of Pappas et al. (2018). Experiments show that simple heuristics do not perform well on the new dataset, and that two neural MRC models that had been tested on BIOREAD perform much better on BIOMRC, indicating that the new dataset is indeed less noisy or at least that its task is more feasible. Non-expert human performance is also higher on the new dataset compared to BIOREAD, and biomedical experts perform even better. We also introduce a new BERT-based MRC model, the best version of which substantially outperforms all other methods tested, reaching or surpassing the accuracy of biomedical experts in some experiments. We make the new dataset available in three different sizes, also releasing our code, and providing a leaderboard.