mlsanityData to 7 Oct 2026

Chain of Hindsight Aligns Language Models with Feedback

arXiv 2302.02676 ↗First posted 6 Feb 2023cs.LG cs.CLcited by 147

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Learning from human preferences is important for language models to match human needs and to align with human and social values. Prior works have achieved remarkable successes by learning from human feedback to understand and follow instructions. Nonetheless, these methods are either founded on hand-picked model generations that are favored by human annotators, rendering them inefficient in terms of data utilization and challenging to apply in general, or they depend on reinforcement learning, which often suffers from imperfect reward functions and relies on extremely challenging optimizations. In this work, we propose a novel technique, Chain of Hindsight, that is easy to optimize and can learn from any form of feedback, regardless of its polarity. Our idea is inspired by how humans learn from extensive feedback presented in the form of languages. We convert all types of feedback into sequences of sentences, which are then used to fine-tune the model, allowing us to take advantage of the language comprehension capabilities of language models. We condition the model on a sequence of model generations paired with feedback. By doing so, the model is trained to generate outputs based on feedback, while learning to identify and correct negative attributes or errors. Applying our method to large language models, we observed that Chain of Hindsight significantly surpasses previous methods in aligning language models with human preferences. We report significant improvements on summarization and dialogue benchmarks, with our approach markedly preferred in human evaluations.

Key facts. Cited by 147 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 64 of its 91 references that are in the map. First posted on arXiv 6 Feb 2023.

Builds on

64 of 91 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  3. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  4. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 58

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  8. 2020GLU Variants Improve Transformercited by 1,724
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
  13. 2018Hierarchical Neural Story Generationcited by 1,182
  14. 2017Hindsight Experience Replaycited by 1,096
  15. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  16. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  17. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  18. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  19. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  20. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  21. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  22. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  23. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  24. 2022Large Language Models Can Self-Improvecited by 656
  25. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  26. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  27. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  28. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  29. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  30. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  31. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 147
  32. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  33. 2022Language Models as Agent Modelscited by 131
  34. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  35. 2019Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Trainingcited by 110
  36. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  37. 2019HEAD-QA: A Healthcare Dataset for Complex Reasoningcited by 100
  38. 2022Evaluating Human-Language Model Interactioncited by 90

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 48; 2024: 56; 2025: 26; 2026: 17.

147 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  4. 2023Teaching Large Language Models to Self-Debugcited by 963
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  6. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  7. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  8. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  9. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  10. 2026Reinforcement Learning via Self-Distillationcited by 285
  11. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  12. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
38 more
  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2023Large Language Model Alignment: A Surveycited by 241
  3. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  4. 2024Compressed Chain of Thought: Efficient Reasoning Through Dense Representationscited by 192
  5. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  6. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  7. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  8. 2023Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sourcescited by 121
  9. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  10. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  11. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  12. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  13. 2024Teams of LLM Agents can Exploit Zero-Day Vulnerabilitiescited by 93
  14. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  15. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
  16. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  17. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  18. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 77
  19. 2023Aligning Large Language Models through Synthetic Feedbackcited by 75
  20. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
  21. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  22. 2024TrustAgent: Towards Safe and Trustworthy LLM-based Agentscited by 62
  23. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 53
  24. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  25. 2024Transfer Q Star: Principled Decoding for LLM Alignmentcited by 46
  26. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
  27. 2024PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Modelscited by 41
  28. 2025A Unified Understanding and Evaluation of Steering Methodscited by 40
  29. 2025Seek in the Dark: Reasoning via Test-Time Instance-Level Policy Gradient in Latent Spacecited by 40
  30. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  31. 2023Thought Propagation: An Analogical Approach to Complex Reasoning with Large Language Modelscited by 37
  32. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignmentcited by 35
  33. 2024Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Modelscited by 35
  34. 2023ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedbackcited by 34
  35. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysiscited by 34
  36. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  37. 2024Do LLM Agents Have Regret? A Case Study in Online Learning and Gamescited by 33
  38. 2024Direct Judgement Preference Optimizationcited by 33

50 most cited of 147 All 147 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followers
  2. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  3. 2023Training Language Models with Language Feedback at Scale
  4. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
  5. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
  6. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  7. 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models
  8. 2020Learning Rewards from Linguistic Feedback