mlsanityData to 7 Oct 2026

Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback

arXiv 2306.00186 ↗First posted 31 May 2023cs.CLcited by 71

Not on the map. Why?

Cited by 23 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite the seeming success of contemporary grounded text generation systems, they often tend to generate factually inconsistent text with respect to their input. This phenomenon is emphasized in tasks like summarization, in which the generated summaries should be corroborated by their source article. In this work, we leverage recent progress on textual entailment models to directly address this problem for abstractive summarization systems. We use reinforcement learning with reference-free, textual entailment rewards to optimize for factual consistency and explore the ensuing trade-offs, as improved consistency may come at the cost of less informative or more extractive summaries. Our results, according to both automatic metrics and human evaluation, show that our method considerably improves the faithfulness, salience, and conciseness of the generated summaries.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 56 of its 72 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

56 of 72 references.

References · 56

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  10. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  11. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  12. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
38 more
  1. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  2. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  3. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  4. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  5. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  6. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  7. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  8. 2018Maximum a Posteriori Policy Optimisationcited by 318
  9. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  10. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  11. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  12. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  13. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  14. 2019A Theory of Regularized Markov Decision Processescited by 279
  15. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  16. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  17. 2015Contextual Markov Decision Processescited by 213
  18. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  19. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  20. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  21. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  22. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  23. 2018Fast Abstractive Summarization with Reinforce-Selected Sentence Rewritingcited by 158
  24. 2018Ranking Sentences for Extractive Summarization with Reinforcement Learningcited by 128
  25. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  26. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  27. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 113
  28. 2020Mirror Descent Policy Optimizationcited by 102
  29. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  30. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 93
  31. 2020Controlled Hallucinations: Learning to Generate Faithfully from Noisy Datacited by 83
  32. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 76
  33. 2018A Study of Reinforcement Learning for Neural Machine Translationcited by 74
  34. 2021Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Featurescited by 74
  35. 2018Multi-Reward Reinforced Summarization with Saliency and Entailmentcited by 73
  36. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 63
  37. 2019Non-Monotonic Sequential Text Generationcited by 55
  38. 2021Machine Translation Decoding beyond Beam Searchcited by 55

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 41; 2025: 14; 2026: 7.

71 papers.

Sort:

By area:

  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  3. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 317
  4. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  5. 2023Nash Learning from Human Feedbackcited by 220
  6. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  7. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 155
  8. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  9. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  10. 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
  11. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 85
  12. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
38 more
  1. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  2. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 49
  3. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  4. 2024Molecular Facts: Desiderata for Decontextualization in LLM Fact Verificationcited by 42
  5. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 42
  6. 2024MusicRL: Aligning Music Generation to Human Preferencescited by 32
  7. 2023Teaching Language Models to Hallucinate Less with Synthetic Taskscited by 31
  8. 2024Learning to Summarize from LLM-generated Feedbackcited by 24
  9. 2025Robust Multi-Objective Preference Alignment with Online DPOcited by 21
  10. 2024Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learningcited by 20
  11. 2024Self-Improving Robust Preference Optimizationcited by 19
  12. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 18
  13. 2024Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generationcited by 15
  14. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  15. 2024Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashioncited by 13
  16. 2024Localizing Factual Inconsistencies in Attributable Text Generationcited by 13
  17. 2024RLPF: Reinforcement Learning from Prediction Feedback for User Summarization with LLMscited by 11
  18. 2024Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewardscited by 11
  19. 2025Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generationcited by 11
  20. 2024Graph-based Confidence Calibration for Large Language Modelscited by 10
  21. 2024Teaching Language Models to Self-Improve by Learning from Language Feedbackcited by 9
  22. 2024Key-Element-Informed sLLM Tuning for Document Summarizationcited by 8
  23. 2024Beyond the Safety Bundle: Auditing the Helpful and Harmless Datasetcited by 8
  24. 2024Model-based Preference Optimization in Abstractive Summarization without Human Feedbackcited by 7
  25. 2024Coarse-to-Fine Highlighting: Reducing Knowledge Hallucination in Large Language Modelscited by 7
  26. 2023Improving Factual Consistency of News Summarization by Contrastive Preference Optimizationcited by 6
  27. 2024Plausible Extractive Rationalization through Semi-Supervised Entailment Signalcited by 6
  28. 2024Multi-Dimensional Optimization for Text Summarization via Reinforcement Learningcited by 6
  29. 2024LLM See, LLM Do: Guiding Data Generation to Target Non-Differentiable Objectivescited by 6
  30. 2025Hallucinate at the Last in Long Response Generation: A Case Study on Long Document Summarizationcited by 6
  31. 2025A Unified Definition of Hallucination: It's The World Model, Stupid!cited by 6
  32. 2024Hallucination Diversity-Aware Active Learning for Text Summarizationcited by 5
  33. 2024Analysis of Plan-based Retrieval for Grounded Text Generationcited by 5
  34. 2023Factual and Personalized Recommendations using Language Models and Reinforcement Learningcited by 4
  35. 2024Transparent and Coherent Procedural Mistake Detectioncited by 4
  36. 2024SEER: Facilitating Structured Reasoning and Explanation via Reinforcement Learningcited by 3
  37. 2024CovRL: Fuzzing JavaScript Engines with Coverage-Guided Reinforcement Learning for LLM-based Mutationcited by 3
  38. 2024uMedSum: A Unified Framework for Advancing Medical Abstractive Summarizationcited by 3

50 most cited of 71 All 71 (JSON)