mlsanityData to 7 Oct 2026

Better Rewards Yield Better Summaries: Learning to Summarise Without References

arXiv 1909.01214 ↗First posted 3 Sep 2019cs.CLcited by 76

Not on the map. Why?

Cited by 25 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reinforcement Learning (RL) based document summarisation systems yield state-of-the-art performance in terms of ROUGE scores, because they directly use ROUGE as the rewards during training. However, summaries with high ROUGE scores often receive low human judgement. To find a better reward function that can guide RL to generate human-appealing summaries, we learn a reward function from human ratings on 2,500 summaries. Our reward function only takes the document and system summary as input. Hence, once trained, it can be used to train RL-based summarisation systems without using any reference summaries. We show that our learned rewards have significantly higher correlation with human ratings than previous approaches. Human evaluation experiments show that, compared to the state-of-the-art supervised-learning systems and ROUGE-as-rewards RL summarisation systems, the RL systems using our learned rewards during training generate summarieswith higher human ratings. The learned reward function and our source code are available at https://github.com/yg211/summary-reward-no-reference.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 9; 2022: 16; 2023: 27; 2024: 16; 2025: 4; 2026: 2.

76 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  5. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  6. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  7. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  8. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  9. 2024The pitfalls of next-token predictioncited by 156
  10. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  11. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  12. 2024Aligning Diffusion Models by Optimizing Human Utilitycited by 99
38 more
  1. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  2. 2022Evaluating Human-Language Model Interactioncited by 90
  3. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  4. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  5. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  6. 2023Strategic Reasoning with Language Modelscited by 64
  7. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  8. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 53
  9. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 42
  10. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Modelscited by 41
  11. 2022The CRINGE Loss: Learning what language not to modelcited by 39
  12. 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
  13. 2023Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversationscited by 38
  14. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Modelscited by 36
  15. 2021Bidimensional Leaderboards: Generate and Evaluate Language Hand in Handcited by 32
  16. 2023Aligning Language Models with Human Preferences via a Bayesian Approachcited by 27
  17. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Modelcited by 25
  18. 2023Generating Summaries with Controllable Readability Levelscited by 24
  19. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  20. 2021A Training-free and Reference-free Summarization Evaluation Metric via Centrality-weighted Relevance and Self-referenced Redundancycited by 21
  21. 2023Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignmentcited by 19
  22. 2022Efficient Unsupervised Sentence Compression by Fine-tuning Transformers with Reinforcement Learningcited by 18
  23. 2024RLVF: Learning from Verbal Feedback without Overgeneralizationcited by 18
  24. 2022Reward Modeling for Mitigating Toxicity in Transformer-based Language Modelscited by 17
  25. 2023NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklistcited by 16
  26. 2022USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translationcited by 15
  27. 2023ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generationcited by 15
  28. 2021Diversity driven Query Rewriting in Search Advertisingcited by 14
  29. 2022MACSum: Controllable Summarization with Mixed Attributescited by 13
  30. 2024SYNFAC-EDIT: Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarizationcited by 12
  31. 2025RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Taskscited by 12
  32. 2025Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RLcited by 11
  33. 2022Innovations in Neural Data-to-text Generation: A Surveycited by 10
  34. 2023Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challengescited by 10
  35. 2024MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipelinecited by 9
  36. 2022Human-in-the-loop Abstractive Dialogue Summarizationcited by 8
  37. 2023Advancements in Scientific Controllable Text Generation Methodscited by 7
  38. 2023Improving Summarization with Human Editscited by 7

50 most cited of 76 All 76 (JSON)