mlsanityData to 7 Oct 2026

On Improving Summarization Factual Consistency from Natural Language Feedback

arXiv 2212.09968 ↗First posted 20 Dec 2022cs.CLcited by 38

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Despite the recent progress in language generation models, their outputs may not always meet user expectations. In this work, we study whether informational feedback in natural language can be leveraged to improve generation quality and user preference alignment. To this end, we consider factual consistency in summarization, the quality that the summary should only contain information supported by the input documents, as the user-expected preference. We collect a high-quality dataset, DeFacto, containing human demonstrations and informational natural language feedback consisting of corrective instructions, edited summaries, and explanations with respect to the factual consistency of the summary. Using our dataset, we study three natural language generation tasks: (1) editing a summary by following the human feedback, (2) generating human feedback for editing the original summary, and (3) revising the initial summary to correct factual errors by generating both the human feedback and edited summary. We show that DeFacto can provide factually consistent human-edited summaries and further insights into summarization factual consistency thanks to its informational natural language feedback. We further demonstrate that fine-tuned language models can leverage our dataset to improve the summary factual consistency, while large language models lack the zero-shot learning ability in our proposed tasks that require controllable text generation.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 52 of its 69 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

52 of 69 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback5 Aug
  2. 2022Training Language Models with Language Feedback29 Apr
  3. 2022Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback12 Apr
  4. 2020Learning to summarize from human feedback2 Sep

Other references · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  9. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  10. 2015Teaching Machines to Read and Comprehendcited by 1,039
  11. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  12. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
36 more
  1. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  2. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  3. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  4. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  5. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  6. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  7. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  8. 2019Levenshtein Transformercited by 197
  9. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  10. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  11. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 128
  12. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  13. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  14. 2020Factual Error Correction for Abstractive Summarization Modelscited by 108
  15. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  16. 2019Encode, Tag, Realize: High-Precision Text Editingcited by 99
  17. 2022PEER: A Collaborative Language Modelcited by 86
  18. 2019Parallel Iterative Edit Models for Local Sequence Transductioncited by 81
  19. 2021Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selectioncited by 75
  20. 2020What Have We Achieved on Text Summarization?cited by 65
  21. 2020Felix: Flexible Text Editing Through Tagging and Insertioncited by 58
  22. 2020Seq2Edits: Sequence Transduction Using Span-level Edit Operationscited by 58
  23. 2022FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarizationcited by 49
  24. 2021LEWIS: Levenshtein Editing for Unsupervised Text Style Transfercited by 40
  25. 2022Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infillingcited by 38
  26. 2022EdiT5: Semi-Autoregressive Text-Editing with T5 Warm-Startcited by 35
  27. 2020Text Editing by Commandcited by 33
  28. 2021Focus Attention: Promoting Faithfulness and Diversity in Summarizationcited by 30
  29. 2022Learning to Revise References for Faithful Summarizationcited by 28
  30. 2022Chain of Explanation: New Prompting Method to Generate Higher Quality Natural Language Explanation for Implicit Hate Speechcited by 25
  31. 2022Text Generation with Text-Editing Modelscited by 21
  32. 2022Improving the Faithfulness of Abstractive Summarization via Entity Coverage Controlcited by 21
  33. 2022An Imitation Learning Curriculum for Text Editing with Non-Autoregressive Modelscited by 20
  34. 2022Improving Factual Consistency in Summarization with Compression-Based Post-Editingcited by 13
  35. 2022DataLab: A Platform for Data Analysis and Interventioncited by 8
  36. 2021Improving Neural Model Performance through Natural Language Feedback on Their Explanationscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 14; 2025: 6; 2026: 2.

38 papers.

Sort:

By area:

  1. 2023Training Language Models with Language Feedback at Scalecited by 125
  2. 2023Shepherd: A Critic for Language Model Generationcited by 96
  3. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  4. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 66
  5. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  6. 2023SummIt: Iterative Text Summarization via ChatGPTcited by 53
  7. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  8. 2023ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviewscited by 34
  9. 2024Learning to Refine with Fine-Grained Natural Language Feedbackcited by 27
  10. 2024Learning to Summarize from LLM-generated Feedbackcited by 24
  11. 2023The Impact of Imperfect XAI on Human-AI Decision-Makingcited by 21
  12. 2023Reasons to Reject? Aligning Language Models with Judgmentscited by 15
26 more
  1. 2023Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challengescited by 13
  2. 2024Localizing Factual Inconsistencies in Attributable Text Generationcited by 13
  3. 2023Investigating Table-to-Text Generation Capabilities of LLMs in Real-World Information Seeking Scenarioscited by 12
  4. 2024GenAudit: Fixing Factual Errors in Language Model Outputs with Evidencecited by 10
  5. 2024AmbigNLG: Addressing Task Ambiguity in Instruction for NLGcited by 10
  6. 2025MAMM-Refine: A Recipe for Improving Faithfulness in Generation with Multi-Agent Collaborationcited by 9
  7. 2023XATU: A Fine-grained Instruction-based Benchmark for Explainable Text Updatescited by 6
  8. 2023Improving Factual Consistency of News Summarization by Contrastive Preference Optimizationcited by 6
  9. 2024Multi-Dimensional Optimization for Text Summarization via Reinforcement Learningcited by 6
  10. 2025Reasoning or Not? A Comprehensive Evaluation of Reasoning LLMs for Dialogue Summarizationcited by 6
  11. 2023USB: A Unified Summarization Benchmark Across Tasks and Domainscited by 5
  12. 2024The Critique of Critiquecited by 5
  13. 2024Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomycited by 5
  14. 2023Personalized Abstractive Summarization by Tri-agent Generation Pipelinecited by 4
  15. 2023P^3SUM: Preserving Author's Perspective in News Summarization with Diffusion Language Modelscited by 4
  16. 2024Learning to Verify Summary Facts with Fine-Grained LLM Feedbackcited by 4
  17. 2025ReFeed: Multi-dimensional Summarization Refinement with Reflective Reasoning on Feedbackcited by 4
  18. 2025RECON: Reasoning with Condensation for Efficient Retrieval-Augmented Generationcited by 3
  19. 2023What if you said that differently?: How Explanation Formats Affect Human Feedback Efficacy and User Perceptioncited by 2
  20. 2024ISQA: Informative Factuality Feedback for Scientific Summarizationcited by 2
  21. 2025Fine-Grained Detection of Context-Grounded Hallucinations Using LLMscited by 1
  22. 2024Improving Factual Error Correction for Abstractive Summarization via Data Distillation and Conditional-generation Clozecited by 0
  23. 2024Zero-shot Factual Consistency Evaluation Across Domainscited by 0
  24. 2025Is the Top Still Spinning? Evaluating Subjectivity in Narrative Understandingcited by 0
  25. 2026Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Rewardcited by 0
  26. 2026Enhancing Factuality through Consensus and Consistency in Summarization Using Minimum Bayes Risk Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Evaluating Factual Consistency of Summaries with Large Language Models
  2. 2026Optimising Factual Consistency in Summarisation via Preference Learning from Multiple Imperfect Metrics
  3. 2022Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency
  4. 2025Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics
  5. 2023Improving Summarization with Human Edits
  6. 2023Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarization
  7. 2025Multi-round, Chain-of-thought Post-editing for Unfaithful Summaries
  8. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization