# K-QA: A Real-World Medical Q&A Benchmark

arXiv: https://arxiv.org/abs/2401.14493 · HTML: https://mlsanity.com/paper/2401.14493/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 51 references that are in the map.
- First posted on arXiv 25 Jan 2024.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering](https://mlsanity.com/paper/2104.08202/) (2021)
- [WiCE: Real-World Entailment for Claims in Wikipedia](https://mlsanity.com/paper/2303.01432/) (2023)
- [mFACE: Multilingual Summarization with Factual Consistency Evaluation](https://mlsanity.com/paper/2212.10622/) (2022)
- [Question-Driven Summarization of Answers to Consumer Health Questions](https://mlsanity.com/paper/2005.09067/) (2020)
- [PropSegmEnt: A Large-Scale Corpus for Proposition-Level Segmentation and Entailment Recognition](https://mlsanity.com/paper/2212.10750/) (2022)

## Built on by (most cited first, 30 of 31)

- [Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions](https://mlsanity.com/paper/2402.18060/) (2024)
- [Large language model validity via enhanced conformal prediction methods](https://mlsanity.com/paper/2406.09714/) (2024)
- [Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks](https://mlsanity.com/paper/2404.00376/) (2024)
- [A Benchmark for Long-Form Medical Question Answering](https://mlsanity.com/paper/2411.09834/) (2024)
- [OLAPH: Improving Factuality in Biomedical Long-form Question Answering](https://mlsanity.com/paper/2405.12701/) (2024)
- [3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark](https://mlsanity.com/paper/2504.13861/) (2025)
- [Geometric Uncertainty for Detecting and Correcting Hallucinations in LLMs](https://mlsanity.com/paper/2509.13813/) (2025)
- [Baichuan-M3: Modeling Clinical Inquiry for Reliable Medical Decision-Making](https://mlsanity.com/paper/2602.06570/) (2026)
- [Structured Outputs Enable General-Purpose LLMs to be Medical Experts](https://mlsanity.com/paper/2503.03194/) (2025)
- [Script Gap: Evaluating LLM Triage on Indian Languages in Native vs Romanized Scripts in a Real World Setting](https://mlsanity.com/paper/2512.10780/) (2025)
- [Putting People in LLMs' Shoes: Generating Better Answers via Question Rewriter](https://mlsanity.com/paper/2408.10573/) (2024)
- [Leveraging Online Data to Enhance Medical Knowledge in a Small Persian Language Model](https://mlsanity.com/paper/2505.16000/) (2025)
- [Healthcare AI GYM for Medical Agents](https://mlsanity.com/paper/2605.02943/) (2026)
- [MedMobile: A mobile-sized language model with clinical capabilities](https://mlsanity.com/paper/2410.09019/) (2024)
- [KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues](https://mlsanity.com/paper/2509.21856/) (2025)
- [ProMedical: Hierarchical Fine-Grained Criteria Modeling for Medical LLM Alignment via Explicit Injection](https://mlsanity.com/paper/2604.08326/) (2026)
- [MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering](https://mlsanity.com/paper/2605.12361/) (2026)
- [Large Language Models for Cancer Communication: Evaluating Linguistic Quality, Safety, and Accessibility in Generative AI](https://mlsanity.com/paper/2505.10472/) (2025)
- [Evaluating Large Language Models for Evidence-Based Clinical Question Answering](https://mlsanity.com/paper/2509.10843/) (2025)
- [CQA-Eval: Designing Reliable Evaluations of Multi-paragraph Clinical QA under Resource Constraints](https://mlsanity.com/paper/2510.10415/) (2025)
- [AdversaRiskQA: An Adversarial Factuality Benchmark for High-Risk Domains](https://mlsanity.com/paper/2601.15511/) (2026)
- [MedInjection-FR: Exploring the Role of Native, Synthetic, and Translated Data in Biomedical Instruction Tuning](https://mlsanity.com/paper/2603.06905/) (2026)
- [PerMedCQA: Benchmarking Large Language Models on Medical Consumer Question Answering in Persian Language](https://mlsanity.com/paper/2505.18331/) (2025)
- [Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA](https://mlsanity.com/paper/2606.19266/) (2026)
- [OpenBioRQ: Unsolved Biomedical Research Questions for Agents](https://mlsanity.com/paper/2606.21959/) (2026)
- [Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering](https://mlsanity.com/paper/2607.06641/) (2026)
- [CANDI: Contextual Alignment for Niche Domains Question Answering](https://mlsanity.com/paper/2607.11891/) (2026)
- [Gaokerena: A Small Persian Medical Language Model Family](https://mlsanity.com/paper/2608.00932/) (2026)
- [PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoning](https://mlsanity.com/paper/2609.04598/) (2026)
- [MedSNIP: Building and Benchmarking Snippet-Level Granularity for Medical Fact Verification](https://mlsanity.com/paper/2609.12884/) (2026)

## Abstract (arXiv)

Ensuring the accuracy of responses provided by large language models (LLMs) is crucial, particularly in clinical settings where incorrect information may directly impact patient health. To address this challenge, we construct K-QA, a dataset containing 1,212 patient questions originating from real-world conversations held on K Health (an AI-driven clinical platform). We employ a panel of in-house physicians to answer and manually decompose a subset of K-QA into self-contained statements. Additionally, we formulate two NLI-based evaluation metrics approximating recall and precision: (1) comprehensiveness, measuring the percentage of essential clinical information in the generated answer and (2) hallucination rate, measuring the number of statements from the physician-curated response contradicted by the LLM answer. Finally, we use K-QA along with these metrics to evaluate several state-of-the-art models, as well as the effect of in-context learning and medically-oriented augmented retrieval schemes developed by the authors. Our findings indicate that in-context learning improves the comprehensiveness of the models, and augmented retrieval is effective in reducing hallucinations. We make K-QA available to to the community to spur research into medically accurate NLP applications.