mlsanityData to 14 Oct 2026

CQA-Eval: Designing Reliable Evaluations of Multi-paragraph Clinical QA under Resource Constraints

arXiv 2510.10415 ↗First posted 12 Oct 2025cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Evaluating multi-paragraph clinical question answering (QA) systems is resource-intensive and challenging: accurate judgments require medical expertise and achieving consistent human judgments over multi-paragraph text is difficult. We introduce CQA-Eval, an evaluation framework and set of evaluation recommendations for limited-resource and high-expertise settings. Based on physician annotations of 300 real patient questions answered by physicians and LLMs, we compare coarse answer-level versus fine-grained sentence-level evaluation over the dimensions of correctness, relevance, and risk disclosure. We find that inter-annotator agreement (IAA) varies by dimension: fine-grained annotation improves agreement on correctness, coarse improves agreement on relevance, and judgments on communicates-risks remain inconsistent. Additionally, annotating only a small subset of sentences can provide reliability comparable to coarse annotations, reducing cost and effort.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 14 of its 28 references that are in the map. First posted on arXiv 12 Oct 2025.

Builds on

14 of 28 references.

References · 14

  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  2. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  3. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 314
  4. 2019A Question-Entailment Approach to Question Answeringcited by 134
  5. 2024Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questionscited by 109
  6. 2024Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Taskscited by 106
  7. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  8. 2023Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queriescited by 82
  9. 2021Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreementcited by 72
  10. 2024K-QA: A Real-World Medical Q&A Benchmarkcited by 31
  11. 2024A Benchmark for Long-Form Medical Question Answeringcited by 25
  12. 2023APPLS: Evaluating Evaluation Metrics for Plain Language Summarizationcited by 15
2 more
  1. 2025A Scalable Framework for Evaluating Health Language Modelscited by 11
  2. 2025MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answeringcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026From Table to Cell: Attention for Better Reasoning with TABALIGNcited by 2