# Question answering systems for health professionals at the point of care -- a systematic review

arXiv: https://arxiv.org/abs/2402.01700 · HTML: https://mlsanity.com/paper/2402.01700/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 123 references that are in the map.
- First posted on arXiv 24 Jan 2024.

## Builds on (most cited first, 30 of 32)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [A Question-Entailment Approach to Question Answering](https://mlsanity.com/paper/1901.08079/) (2019)
- [BioMegatron: Larger Biomedical Domain Language Model](https://mlsanity.com/paper/2010.06060/) (2020)
- [Biomedical Question Answering: A Survey of Approaches and Challenges](https://mlsanity.com/paper/2102.05281/) (2021)
- [Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3 (with Varying Success)](https://mlsanity.com/paper/2305.06299/) (2023)
- [Diving Deep into Modes of Fact Hallucinations in Dialogue Systems](https://mlsanity.com/paper/2301.04449/) (2023)
- [Neural Domain Adaptation for Biomedical Question Answering](https://mlsanity.com/paper/1706.03610/) (2017)
- [Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies](https://mlsanity.com/paper/2102.08366/) (2021)
- [Unsupervised Domain Adaptation of Language Models for Reading Comprehension](https://mlsanity.com/paper/1911.10768/) (2019)
- [Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERT](https://mlsanity.com/paper/2107.04374/) (2021)
- [Sequence tagging for biomedical extractive question answering](https://mlsanity.com/paper/2104.07535/) (2021)
- [F1 is Not Enough! Models and Evaluation Towards User-Centered Explainable Question Answering](https://mlsanity.com/paper/2010.06283/) (2020)
- [What Would it Take to get Biomedical QA Systems into Practice?](https://mlsanity.com/paper/2109.10415/) (2021)
- [Using Centroids of Word Embeddings and Word Mover's Distance for Biomedical Document Retrieval in Question Answering](https://mlsanity.com/paper/1608.03905/) (2016)
- [Adapting and evaluating a deep learning language model for clinical why-question answering](https://mlsanity.com/paper/1911.05604/) (2019)
- [Task-adaptive Pre-training of Language Models with Word Embedding Regularization](https://mlsanity.com/paper/2109.08354/) (2021)
- [Data Augmentation for Biomedical Factoid Question Answering](https://mlsanity.com/paper/2204.04711/) (2022)
- [Neural Question Answering at BioASQ 5B](https://mlsanity.com/paper/1706.08568/) (2017)
- [CLINIQA: A Machine Intelligence Based Clinical Question Answering System](https://mlsanity.com/paper/1805.05927/) (2018)
- [Causal Perception in Question-Answering Systems](https://mlsanity.com/paper/2012.14477/) (2020)

## Built on by (most cited first, 3 of 3)

- [RealMedQA: A pilot biomedical question answering dataset containing realistic clinical questions](https://mlsanity.com/paper/2408.08624/) (2024)
- [Optimized Biomedical Question-Answering Services with LLM and Multi-BERT Integration](https://mlsanity.com/paper/2410.12856/) (2024)
- [ECG-LLM -- training and evaluation of domain-specific large language models for electrocardiography](https://mlsanity.com/paper/2510.18339/) (2025)

## Abstract (arXiv)

Objective: Question answering (QA) systems have the potential to improve the quality of clinical care by providing health professionals with the latest and most relevant evidence. However, QA systems have not been widely adopted. This systematic review aims to characterize current medical QA systems, assess their suitability for healthcare, and identify areas of improvement. Materials and methods: We searched PubMed, IEEE Xplore, ACM Digital Library, ACL Anthology and forward and backward citations on 7th February 2023. We included peer-reviewed journal and conference papers describing the design and evaluation of biomedical QA systems. Two reviewers screened titles, abstracts, and full-text articles. We conducted a narrative synthesis and risk of bias assessment for each study. We assessed the utility of biomedical QA systems. Results: We included 79 studies and identified themes, including question realism, answer reliability, answer utility, clinical specialism, systems, usability, and evaluation methods. Clinicians' questions used to train and evaluate QA systems were restricted to certain sources, types and complexity levels. No system communicated confidence levels in the answers or sources. Many studies suffered from high risks of bias and applicability concerns. Only 8 studies completely satisfied any criterion for clinical utility, and only 7 reported user evaluations. Most systems were built with limited input from clinicians. Discussion: While machine learning methods have led to increased accuracy, most studies imperfectly reflected real-world healthcare information needs. Key research priorities include developing more realistic healthcare QA datasets and considering the reliability of answer sources, rather than merely focusing on accuracy.