mlsanityData to 14 Oct 2026

RealMedQA: A pilot biomedical question answering dataset containing realistic clinical questions

arXiv 2408.08624 ↗First posted 16 Aug 2024cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Clinical question answering systems have the potential to provide clinicians with relevant and timely answers to their questions. Nonetheless, despite the advances that have been made, adoption of these systems in clinical settings has been slow. One issue is a lack of question-answering datasets which reflect the real-world needs of health professionals. In this work, we present RealMedQA, a dataset of realistic clinical questions generated by humans and an LLM. We describe the process for generating and verifying the QA pairs and assess several QA models on BioASQ and RealMedQA to assess the relative difficulty of matching answers to questions. We show that the LLM is more cost-efficient for generating "ideal" QA pairs. Additionally, we achieve a lower lexical similarity between questions and answers than BioASQ which provides an additional challenge to the top two QA models, as per the results. We release our code and our dataset publicly to encourage further research.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 18 of its 32 references that are in the map. First posted on arXiv 16 Aug 2024.

Builds on

18 of 32 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  8. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  9. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  10. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  11. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  12. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
6 more
  1. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  2. 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 135
  3. 2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answeringcited by 127
  4. 2022Language Model Cascadescited by 107
  5. 2021What Would it Take to get Biomedical QA Systems into Practice?cited by 7
  6. 2024Question answering systems for health professionals at the point of care -- a systematic reviewcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  2. 2026MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answeringcited by 2
  3. 2026Quantifying Hallucinations in Language Language Models on Medical Textbookscited by 0
  4. 2026EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluationcited by 0
  5. 2026Can I Take Another Dose? Evaluating LLM Decision-Making Under Temporal Uncertainty in OTC Dosing QAcited by 0