# FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence

arXiv: https://arxiv.org/abs/2402.11456 · HTML: https://mlsanity.com/paper/2402.11456/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 52 references that are in the map.
- First posted on arXiv 18 Feb 2024.

## Builds on (most cited first, 27 of 27)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [AlignScore: Evaluating Factual Consistency with a Unified Alignment Function](https://mlsanity.com/paper/2305.16739/) (2023)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [QuestEval: Summarization Asks for Fact-based Evaluation](https://mlsanity.com/paper/2103.12693/) (2021)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [Fine-tuning Language Models for Factuality](https://mlsanity.com/paper/2311.08401/) (2023)
- [Annotating and Modeling Fine-grained Factuality in Summarization](https://mlsanity.com/paper/2104.04302/) (2021)
- [Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors](https://mlsanity.com/paper/2205.12854/) (2022)
- [ChatGPT as a Factual Inconsistency Evaluator for Text Summarization](https://mlsanity.com/paper/2303.15621/) (2023)
- [Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration](https://mlsanity.com/paper/2305.14324/) (2023)
- [Inferring Which Medical Treatments Work from Reports of Clinical Trials](https://mlsanity.com/paper/1904.01606/) (2019)
- [MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics](https://mlsanity.com/paper/2010.03636/) (2020)
- [Evidence Inference 2.0: More Data, Better Models](https://mlsanity.com/paper/2005.04177/) (2020)
- [Summarizing, Simplifying, and Synthesizing Medical Evidence Using GPT-3 (with Varying Success)](https://mlsanity.com/paper/2305.06299/) (2023)
- [Elaborative Simplification: Content Addition and Explanation Generation in Text Simplification](https://mlsanity.com/paper/2010.10035/) (2020)
- [InfoLossQA: Characterizing and Recovering Information Loss in Text Simplification](https://mlsanity.com/paper/2401.16475/) (2024)
- [Trialstreamer: Mapping and Browsing Medical Evidence in Real-Time](https://mlsanity.com/paper/2005.10865/) (2020)
- [Thresh: A Unified, Customizable and Deployable Platform for Fine-Grained Text Evaluation](https://mlsanity.com/paper/2308.06953/) (2023)

## Built on by (most cited first, 18 of 18)

- [Accelerating Clinical Evidence Synthesis with Large Language Models](https://mlsanity.com/paper/2406.17755/) (2024)
- [A Survey of Automatic Hallucination Evaluation on Natural Language Generation](https://mlsanity.com/paper/2404.12041/) (2024)
- [Standardizing Intelligence: Aligning Generative AI for Regulatory and Operational Compliance](https://mlsanity.com/paper/2503.04736/) (2025)
- [Can Language Models Evaluate Human Written Text? Case Study on Korean Student Writing for Education](https://mlsanity.com/paper/2407.17022/) (2024)
- [PlainQAFact: Retrieval-augmented Factual Consistency Evaluation Metric for Biomedical Plain Language Summarization](https://mlsanity.com/paper/2503.08890/) (2025)
- [TracSum: A New Benchmark for Aspect-Based Summarization with Sentence-Level Traceability in Medical Domain](https://mlsanity.com/paper/2508.13798/) (2025)
- [Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?](https://mlsanity.com/paper/2502.07963/) (2025)
- [Leveraging language models for summarizing mental state examinations: A comprehensive evaluation and dataset release](https://mlsanity.com/paper/2403.20145/) (2024)
- [How Private are Language Models in Abstractive Summarization?](https://mlsanity.com/paper/2412.12040/) (2024)
- [PERCS: Persona-Guided Controllable Biomedical Summarization Dataset](https://mlsanity.com/paper/2512.03340/) (2025)
- [Mitigating hallucinations in healthcare LLMs with granular fact-checking and domain-specific adaptation](https://mlsanity.com/paper/2512.16189/) (2025)
- [Do We Need Bigger Models for Science? Task-Aware Retrieval with Small Language Models](https://mlsanity.com/paper/2604.01965/) (2026)
- [Assessing Automated Fact-Checking for Medical LLM Responses with Knowledge Graphs](https://mlsanity.com/paper/2511.12817/) (2025)
- [MAPLE: Medical Aspect-Based Summarization with Phrase-Level Evidence](https://mlsanity.com/paper/2601.03418/) (2026)
- [Gavel: Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization](https://mlsanity.com/paper/2601.04424/) (2026)
- [AgenticSum: An Agentic Inference-Time Framework for Faithful Clinical Text Summarization](https://mlsanity.com/paper/2602.20040/) (2026)
- [DWTSumm: Discrete Wavelet Transform for Document Summarization](https://mlsanity.com/paper/2604.21070/) (2026)
- [Does Finetuning with Scientific Data Increase Hallucinations? A Multi-domain Factuality Evaluation of LLMs](https://mlsanity.com/paper/2606.21359/) (2026)

## Abstract (arXiv)

Plain language summarization with LLMs can be useful for improving textual accessibility of technical content. But how factual are these summaries in a high-stakes domain like medicine? This paper presents FactPICO, a factuality benchmark for plain language summarization of medical texts describing randomized controlled trials (RCTs), which are the basis of evidence-based medicine and can directly inform patient treatment. FactPICO consists of 345 plain language summaries of RCT abstracts generated from three LLMs (i.e., GPT-4, Llama-2, and Alpaca), with fine-grained evaluation and natural language rationales from experts. We assess the factuality of critical elements of RCTs in those summaries: Populations, Interventions, Comparators, Outcomes (PICO), as well as the reported findings concerning these. We also evaluate the correctness of the extra information (e.g., explanations) added by LLMs. Using FactPICO, we benchmark a range of existing factuality metrics, including the newly devised ones based on LLMs. We find that plain language summarization of medical evidence is still challenging, especially when balancing between simplicity and factuality, and that existing metrics correlate poorly with expert judgments on the instance level.