mlsanityData to 14 Oct 2026

Improving the Evaluation of Biomedical Question Answering with Few Human Preference Signals

arXiv 2610.16008 ↗First posted 12 Oct 2026cs.CL cs.LGcited by 0

On the map: Retrieval-augmented generation → Medical and biomedical RAG

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems are increasingly used for biomedical Question Answering (QA). However, evaluating their effectiveness remains challenging due to the lack of reliable reference resources and the frequent misalignment between automatic evaluation metrics and human judgments. One practical question is how evaluation results should be used to guide development decisions, especially when they correlate weakly with human judgments and may even lead to different conclusions. We explore how to use a small amount of human preference signals to calibrate direct assessment evaluation metrics, and show that calibrated metrics consistently outperform baselines that either treat all metrics equally or rely on a single best metric across multiple datasets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. Builds on 31 of its 52 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

31 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024BioRAG: A RAG-LLM Framework for Biological Question Reasoning2 Aug

Other references · 30

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  4. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  5. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  6. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  7. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  8. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 301
  9. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
  10. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  11. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  12. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 211
18 more
  1. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 156
  2. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 151
  3. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 137
  4. 2023GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Informationcited by 117
  5. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 86
  6. 2018RankME: Reliable Human Ratings for Natural Language Generationcited by 69
  7. 2024A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chainscited by 65
  8. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Taskscited by 24
  9. 2024Evaluation of RAG Metrics for Question Answering in the Telecom Domaincited by 20
  10. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competitioncited by 18
  11. 2025Evaluating LLM Metrics Through Real-World Capabilitiescited by 18
  12. 2025YESciEval: Robust LLM-as-a-Judge for Scientific Question Answeringcited by 16
  13. 2025What Matters in Data for DPO?cited by 12
  14. 2024GP-GPT: Large Language Model for Gene-Phenotype Mappingcited by 11
  15. 2024Geneverse: A collection of Open-source Multimodal Large Language Models for Genomic and Proteomic Researchcited by 9
  16. 2025Bridging Human and LLM Judgments: Understanding and Narrowing the Gapcited by 7
  17. 2025MedTrust-RAG: Evidence Verification and Trust Alignment for Biomedical Question Answeringcited by 2
  18. 2025The illusion of a perfect metric: Why evaluating AI's words is harder than it lookscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient and Reproducible Biomedical Question Answering using Retrieval Augmented Generation
  2. 2026When Retrieval Doesn't Help: A Large-Scale Study of Biomedical RAG
  3. 2026Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study
  4. 2025RAG-BioQA: A Retrieval-Augmented Generation Framework for Long-Form Biomedical Question Answering
  5. 2024Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answering
  6. 2026RAG-X: Systematic Diagnosis of Retrieval-Augmented Generation for Medical Question Answering
  7. 2025ASTRID -- An Automated and Scalable TRIaD for the Evaluation of RAG-based Clinical Question Answering Systems
  8. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems