mlsanityData to 7 Oct 2026

A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization

arXiv 2303.03948 ↗First posted 7 Mar 2023cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Long-form clinical summarization of hospital admissions has real-world significance because of its potential to help both clinicians and patients. The faithfulness of summaries is critical to their safe usage in clinical settings. To better understand the limitations of abstractive systems, as well as the suitability of existing evaluation metrics, we benchmark faithfulness metrics against fine-grained human annotations for model-generated summaries of a patient's Brief Hospital Course. We create a corpus of patient hospital admissions and summaries for a cohort of HIV patients, each with complex medical histories. Annotators are presented with summaries and source notes, and asked to categorize manually highlighted summary elements (clinical entities like conditions and medications as well as actions like "following up") into one of three categories: ``Incorrect,'' ``Missing,'' and ``Not in Notes.'' We meta-evaluate a broad set of proposed faithfulness metrics and, across metrics, explore the importance of domain adaptation (e.g. the impact of in-domain pre-training and metric fine-tuning), the use of source-summary alignments, and the effects of distilling a single metric from an ensemble of pre-existing metrics. Off-the-shelf metrics with no exposure to clinical text correlate well yet overly rely on summary extractiveness. As a practical guide to long-form clinical narrative summarization, we find that most metrics correlate best to human judgments when provided with one summary sentence at a time and a minimal set of relevant source context.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 62 of its 98 references that are in the map. First posted on arXiv 7 Mar 2023.

Builds on

62 of 98 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
38 more
  1. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  2. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  4. 2015Teaching Machines to Read and Comprehendcited by 1,039
  5. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  6. 2019Publicly Available Clinical BERT Embeddingscited by 995
  7. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  8. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  9. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  10. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  11. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  12. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  13. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  14. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  15. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  16. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  17. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  18. 2020Self-Alignment Pretraining for Biomedical Entity Representationscited by 233
  19. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  20. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  21. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 149
  22. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  23. 2019Hierarchical Transformers for Multi-Document Summarizationcited by 132
  24. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  25. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 128
  26. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  27. 2019Assessing The Factual Accuracy of Generated Textcited by 103
  28. 2023Do We Still Need Clinical Language Models?cited by 97
  29. 2019Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reportscited by 96
  30. 2021Planning with Learned Entity Prompts for Abstractive Summarizationcited by 75
  31. 2020What Have We Achieved on Text Summarization?cited by 65
  32. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  33. 2020Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentationcited by 53
  34. 2022Investigating Efficiently Extending Transformers for Long Input Summarizationcited by 43
  35. 2022FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representationscited by 41
  36. 2021What's in a Summary? Laying the Groundwork for Advances in Hospital-Course Summarizationcited by 35
  37. 2019Scoring Sentence Singletons and Pairs for Abstractive Summarizationcited by 34
  38. 2022Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarizationcited by 34

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 7; 2025: 0; 2026: 1.

14 papers.

Sort:

  1. 2023From Sparse to Dense: GPT-4 Summarization with Chain of Density Promptingcited by 52
  2. 2023NoteChat: A Dataset of Synthetic Doctor-Patient Conversations Conditioned on Clinical Notescited by 48
  3. 2023Less is More for Long Document Summary Evaluation by LLMscited by 28
  4. 2024FineRadScore: A Radiology Report Line-by-Line Evaluation Technique Generating Corrections with Severity Scorescited by 18
  5. 2023README: Bridging Medical Jargon and Lay Understanding for Patient Education through Data-Centric NLPcited by 17
  6. 2024GenAudit: Fixing Factual Errors in Language Model Outputs with Evidencecited by 10
  7. 2023Improving Summarization with Human Editscited by 7
  8. 2024A Survey of Automatic Hallucination Evaluation on Natural Language Generationcited by 7
  9. 2024A Data-Centric Approach To Generate Faithful and High Quality Patient Summaries with Large Language Modelscited by 6
  10. 2024SPEER: Sentence-Level Planning of Long Clinical Summaries via Embedded Entity Retrievalcited by 5
  11. 2024Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questionscited by 5
  12. 2024Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Reviewcited by 3
2 more
  1. 2023Do Physicians Know How to Prompt? The Need for Automatic Prompt Optimization Help in Clinical Note Generationcited by 2
  2. 2026Reverse Probing: Supervised Token-level Uncertainty Quantification for Large Language Models in Clinical Textcited by 0