mlsanityData to 7 Oct 2026

Recursively Summarizing Books with Human Feedback

arXiv 2109.10862 ↗First posted 22 Sep 2021cs.CL cs.AI cs.LGcited by 287

Not on the map. Why?

Cited by 107 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A major challenge for scaling machine learning is training models to perform tasks that are very difficult or time-consuming for humans to evaluate. We present progress on this problem on the task of abstractive summarization of entire fiction novels. Our method combines learning from human feedback with recursive task decomposition: we use models trained on smaller parts of the task to assist humans in giving feedback on the broader task. We collect a large volume of demonstrations and comparisons from human labelers, and fine-tune GPT-3 using behavioral cloning and reward modeling to do summarization recursively. At inference time, the model first summarizes small sections of the book and then recursively summarizes these summaries to produce a summary of the entire book. Our human labelers are able to supervise and evaluate the models quickly, despite not having read the entire books themselves. Our resulting model generates sensible summaries of entire books, even matching the quality of human-written summaries in a few cases (~5% of books). We achieve state-of-the-art results on the recent BookSum dataset for book-length summarization. A zero-shot question-answering model using these summaries achieves state-of-the-art results on the challenging NarrativeQA benchmark for answering questions about books and movie scripts. We release datasets of samples from our model.

Key facts. Cited by 287 papers in the map (data through 7 Oct 2026). Builds on 50 of its 89 references that are in the map. First posted on arXiv 22 Sep 2021.

Builds on

50 of 89 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2018Hierarchical Neural Story Generationcited by 1,182
  10. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  11. 2019Text Summarization with Pretrained Encoderscited by 553
  12. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
38 more
  1. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  2. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  3. 2018AI safety via debatecited by 384
  4. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  5. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  6. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  7. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  8. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  9. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  10. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  11. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  12. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  13. 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 155
  14. 2018Supervising strong learners by amplifying weak expertscited by 139
  15. 2019Hierarchical Transformers for Multi-Document Summarizationcited by 132
  16. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 120
  17. 2020Unsupervised Question Decomposition for Question Answeringcited by 118
  18. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  19. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  20. 2019On Extractive and Abstractive Neural Document Summarization with Transformer Language Modelscited by 83
  21. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 76
  22. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 71
  23. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  24. 2019Extractive Summarization of Long Documents by Combining Global and Local Contextcited by 59
  25. 2020Multi-document Summarization via Deep Learning Techniques: A Surveycited by 58
  26. 2020Generating Fact Checking Briefscited by 50
  27. 2020Hidden Incentives for Auto-Induced Distributional Shiftcited by 42
  28. 2019Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluatorscited by 34
  29. 2017A Supervised Approach to Extractive Summarisation of Scientific Paperscited by 33
  30. 2018Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedbackcited by 32
  31. 2020Exploring Content Selection in Summarization of Novel Chapterscited by 29
  32. 2020Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Modelscited by 27
  33. 2019Finding Generalizable Evidence by Learning to Convince Q&A Modelscited by 25
  34. 2013New Alignment Methods for Discriminative Book Summarizationcited by 18
  35. 2020SEAL: Segment-wise Extractive-Abstractive Long-form Text Summarizationcited by 15
  36. 2021ReadTwice: Reading Very Large Documents with Memoriescited by 14
  37. 2020On Generating Extended Summaries of Long Documentscited by 10
  38. 2020Frustratingly Hard Evidence Retrieval for QA Over Bookscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 35; 2023: 91; 2024: 85; 2025: 46; 2026: 22.

287 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  7. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  8. 2022Red Teaming Language Models with Language Modelscited by 983
  9. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  10. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  12. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
38 more
  1. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  3. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  4. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  5. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  6. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  7. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  8. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  9. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  10. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  11. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  12. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  13. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  14. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  15. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  16. 2023Augmented Language Models: a Surveycited by 398
  17. 2022Self-critiquing models for assisting human evaluatorscited by 372
  18. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  19. 2025Memory in the Age of AI Agentscited by 285
  20. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  21. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  22. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  23. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  24. 2023Challenges and Applications of Large Language Modelscited by 251
  25. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  26. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  27. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  28. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  29. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  30. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  31. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  32. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  33. 2022Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionalscited by 191
  34. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  35. 2023Mitigating the Alignment Tax of RLHFcited by 181
  36. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  37. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  38. 2024How to Train Long-Context Language Models (Effectively)cited by 141

50 most cited of 287 All 287 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning to Summarize from LLM-generated Feedback
  2. 2026Towards Human-Level Book-Writing Capability
  3. 2024Model-based Preference Optimization in Abstractive Summarization without Human Feedback
  4. 2023Improving Summarization with Human Edits
  5. 2022Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback
  6. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization
  7. 2022Training Language Models with Language Feedback
  8. 2024LOCOST: State-Space Models for Long Document Abstractive Summarization