mlsanityData to 14 Oct 2026

Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback

arXiv 2204.05512 ↗First posted 12 Apr 2022cs.AIcited by 13

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

For summarization, human preference is critical to tame outputs of the summarizer in favor of human interests, as ground-truth summaries are scarce and ambiguous. Practical settings require dynamic exchanges between human and AI agent wherein feedback is provided in an online manner, a few at a time. In this paper, we introduce a new framework to train summarization models with preference feedback interactively. By properly leveraging offline data and a novel reward model, we improve the performance regarding ROUGE scores and sample-efficiency. Our experiments on three various datasets confirm the benefit of the proposed framework in active, few-shot and online settings of preference learning.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 28 references that are in the map. First posted on arXiv 12 Apr 2022.

Builds on

20 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Learning to summarize from human feedback2 Sep

Other references · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2019Text Summarization with Pretrained Encoderscited by 556
  7. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  8. 2021A Survey of Human-in-the-loop for Machine Learningcited by 215
  9. 2018Ranking Sentences for Extractive Summarization with Reinforcement Learningcited by 128
  10. 2019BillSum: A Corpus for Automatic Summarization of US Legislationcited by 117
  11. 2018Abstractive Summarization of Reddit Posts with Multi-level Memory Networkscited by 102
  12. 2018Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Studycited by 90
7 more
  1. 2018Multi-Reward Reinforced Summarization with Saliency and Entailmentcited by 73
  2. 2017Human-in-the-loop Artificial Intelligencecited by 40
  3. 2021Survey on reinforcement learning for language processingcited by 31
  4. 2019Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarizationcited by 24
  5. 2021Robust Deep Reinforcement Learning for Extractive Legal Summarizationcited by 10
  6. 2018APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learningcited by 9
  7. 2018The Rule of Three: Abstractive Text Summarization in Three Bullet Pointscited by 1

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 5; 2024: 5; 2025: 1.

13 papers.

Sort:

By area:

  1. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  2. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 87
  3. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  4. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 54
  5. 2022On Improving Summarization Factual Consistency from Natural Language Feedbackcited by 38
  6. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 18
  7. 2025MAMM-Refine: A Recipe for Improving Faithfulness in Generation with Multi-Agent Collaborationcited by 9
  8. 2024PerSEval: Assessing Personalization in Text Summarizerscited by 3
  9. 2024Aligning Model Evaluations with Human Preferences: Mitigating Token Count Bias in Language Model Assessmentscited by 3
  10. 2024Are Large Language Models In-Context Personalized Summarizers? Get an iCOPERNICUS Test Done!cited by 2
  11. 2023SPEC: Summary Preference Decomposition for Low-Resource Abstractive Summarizationcited by 1
  12. 2023When Giant Language Brains Just Aren't Enough! Domain Pizzazz with Knowledge Sparkle Dustcited by 1
1 more
  1. 2022Improving Document Image Understanding with Reinforcement Finetuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Model-based Preference Optimization in Abstractive Summarization without Human Feedback
  2. 2025Learning to summarize user information for personalized reinforcement learning from human feedback
  3. 2023The Impact of Preference Agreement in Reinforcement Learning from Human Feedback: A Case Study in Summarization
  4. 2024Learning to Summarize from LLM-generated Feedback
  5. 2024How Much Annotation is Needed to Compare Summarization Models?
  6. 2025Learning from Self Critique and Refinement for Faithful LLM Summarization
  7. 2024AlignSum: Data Pyramid Hierarchical Fine-tuning for Aligning with Human Summarization Preference
  8. 2022Towards Abstractive Timeline Summarisation using Preference-based Reinforcement Learning