mlsanityData to 14 Oct 2026

PlainQAFact: Retrieval-augmented Factual Consistency Evaluation Metric for Biomedical Plain Language Summarization

arXiv 2503.08890 ↗First posted 11 Mar 2025cs.CLcited by 4

On the map: Retrieval-augmented generation → Medical and biomedical RAG

Abstract (arXiv)

Hallucinated outputs from large language models (LLMs) pose risks in the medical domain, especially for lay audiences making health-related decisions. Existing automatic factual consistency evaluation methods, such as entailment- and question-answering (QA) -based, struggle with plain language summarization (PLS) due to elaborative explanation phenomenon, which introduces external content (e.g., definitions, background, examples) absent from the scientific abstract to enhance comprehension. To address this, we introduce PlainQAFact, an automatic factual consistency evaluation metric trained on a fine-grained, human-annotated dataset PlainFact, for evaluating factual consistency of both source-simplified and elaborately explained sentences. PlainQAFact first classifies sentence type, then applies a retrieval-augmented QA scoring method. Empirical results show that existing evaluation metrics fail to evaluate the factual consistency in PLS, especially for elaborative explanations, whereas PlainQAFact consistently outperforms them across all evaluation settings. We further analyze PlainQAFact's effectiveness across external knowledge sources, answer extraction strategies, answer overlap measures, and document granularity levels, refining its overall factual consistency assessment. Taken together, our work presents a sentence-aware, retrieval-augmented metric targeted at elaborative explanations in biomedical PLS tasks, providing the community with both a new benchmark and a practical evaluation tool to advance reliable and safe plain language communication in the medical domain. PlainQAFact and PlainFact are available at: https://github.com/zhiwenyou103/PlainQAFact

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. Builds on 37 of its 61 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

37 of 61 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025MedBioLM: Optimizing Medical and Biological QA with Fine-Tuned Large Language Models and Retrieval-Augmented Generation5 Feb
  2. 2024Benchmarking Retrieval-Augmented Generation for Medicine20 Feb

Other references · 35

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2017Billion-scale similarity search with GPUscited by 2,497
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  8. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  9. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  10. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
  11. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 257
  12. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
23 more
  1. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  2. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  3. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  4. 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 185
  5. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 147
  6. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  7. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  8. 2024Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answeringcited by 96
  9. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 88
  10. 2022Generating Scientific Claims for Zero-Shot Scientific Fact Checkingcited by 59
  11. 2020Automated Lay Language Summarization of Biomedical Scientific Reviewscited by 46
  12. 2020MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metricscited by 42
  13. 2020Elaborative Simplification: Content Addition and Explanation Generation in Text Simplificationcited by 27
  14. 2019HighRES: Highlight-based Reference-less Evaluation of Summarizationcited by 21
  15. 2022Retrieval augmentation of large language models for lay language generationcited by 21
  16. 2024FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidencecited by 18
  17. 2024Beyond Binary Gender Labels: Revealing Gender Biases in LLMs through Gender-Neutral Name Predictionscited by 17
  18. 2023APPLS: Evaluating Evaluation Metrics for Plain Language Summarizationcited by 15
  19. 2023PromptRank: Unsupervised Keyphrase Extraction Using Promptcited by 13
  20. 2023Personalized Jargon Identification for Enhanced Interdisciplinary Communicationcited by 12
  21. 2022Evaluating and Improving Factuality in Multimodal Abstractive Summarizationcited by 6
  22. 2024Using Similarity to Evaluate Factual Consistency in Summariescited by 5
  23. 2024SciPrompt: Knowledge-augmented Prompting for Fine-grained Categorization of Scientific Topicscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Improving Clinical Diagnosis with Counterfactual Multi-Agent Reasoningcited by 5
  2. 2026Five Queries Are Enough: Query-Efficient and Surrogate-Free Membership Inference Attacks on RAG via Entailmentcited by 4
  3. 2025Explainable Semantic Text Relations: A Question-Answering Framework for Comparing Document Contentcited by 1
  4. 2026ReLay: Personalized LLM-Generated Plain-Language Summaries for Better Understanding, but at What Cost?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Clinfo.ai: An Open-Source Retrieval-Augmented Large Language Model System for Answering Medical Questions using Scientific Literature
  2. 2025AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment
  3. 2026BioACE: An Automated Framework for Biomedical Answer and Citation Evaluations
  4. 2025Evaluating Large Language Models for Evidence-Based Clinical Question Answering
  5. 2024Rethinking Scientific Summarization Evaluation: Grounding Explainable Metrics on Facet-aware Benchmark
  6. 2022Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency
  7. 2025RGAR: Recurrence Generation-augmented Retrieval for Factual-aware Medical Question Answering
  8. 2025Improving Reliability and Explainability of Medical Question Answering through Atomic Fact Checking in Retrieval-Augmented LLMs