mlsanityData to 7 Oct 2026

The Wisdom of Hindsight Makes Language Models Better Instruction Followers

arXiv 2302.05206 ↗First posted 10 Feb 2023cs.CL cs.AIcited by 59

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning has seen wide success in finetuning large language models to better align with instructions via human feedback. The so-called algorithm, Reinforcement Learning with Human Feedback (RLHF) demonstrates impressive performance on the GPT series models. However, the underlying Reinforcement Learning (RL) algorithm is complex and requires an additional training pipeline for reward and value networks. In this paper, we consider an alternative approach: converting feedback to instruction by relabeling the original one and training the model for better alignment in a supervised manner. Such an algorithm doesn't require any additional parameters except for the original language model and maximally reuses the pretraining pipeline. To achieve this, we formulate instruction alignment problem for language models as a goal-reaching problem in decision making. We propose Hindsight Instruction Relabeling (HIR), a novel algorithm for aligning language models with instructions. The resulting two-stage algorithm shed light to a family of reward-free approaches that utilize the hindsightly relabeled instructions based on feedback. We evaluate the performance of HIR extensively on 12 challenging BigBench reasoning tasks and show that HIR outperforms the baseline algorithms and is comparable to or even surpasses supervised finetuning.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 54 references that are in the map. First posted on arXiv 10 Feb 2023.

Builds on

46 of 54 references.

Closest ancestors · 5

Same direction, newest first

  1. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  2. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  3. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  4. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 41

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
29 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  7. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  9. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  10. 2017Hindsight Experience Replaycited by 1,096
  11. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  12. 2021Ethical and social risks of harm from Language Modelscited by 957
  13. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  14. 2021GPT Understands, Toocited by 715
  15. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  16. 2022Large Language Models Can Self-Improvecited by 656
  17. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  18. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  19. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  20. 2021True Few-Shot Learning with Language Modelscited by 343
  21. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  22. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  23. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  24. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  25. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  26. 2022Online Decision Transformercited by 194
  27. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  28. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  29. 2018Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedbackcited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 15; 2025: 8; 2026: 15.

59 papers.

Sort:

By area:

  1. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  2. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  3. 2023Cognitive Architectures for Language Agentscited by 422
  4. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  5. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  6. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  7. 2026Reinforcement Learning via Self-Distillationcited by 285
  8. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  9. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  10. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  11. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  12. 2023Learning Performance-Improving Code Editscited by 103
38 more
  1. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  2. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
  3. 2023Aligning Large Language Models with Human Preferences through Representation Engineeringcited by 71
  4. 2023Evaluating the Zero-shot Robustness of Instruction-tuned Language Modelscited by 63
  5. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  6. 2024Can AI Assistants Know What They Don't Know?cited by 47
  7. 2024SpeechAlign: Aligning Speech Generation to Human Preferencescited by 45
  8. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 36
  9. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignmentcited by 35
  10. 2026Dynamic Dual-Granularity Skill Bank for Agentic RLcited by 33
  11. 2023Teaching Language Models to Self-Improve through Interactive Demonstrationscited by 26
  12. 2023LeTI: Learning to Generate from Textual Interactionscited by 25
  13. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learningcited by 25
  14. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  15. 2023Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyondcited by 20
  16. 2023Reasons to Reject? Aligning Language Models with Judgmentscited by 15
  17. 2023Can Language Model Moderators Improve the Health of Online Discourse?cited by 12
  18. 2023Learning from Mistakes via Cooperative Study Assistant for Large Language Modelscited by 9
  19. 2024Soft Preference Optimization: Aligning Language Models to Expert Distributionscited by 9
  20. 2024E2CL: Exploration-based Error Correction Learning for Embodied Agentscited by 9
  21. 2026HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillationcited by 8
  22. 2024Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Explorationcited by 7
  23. 2024Generalizing Reward Modeling for Out-of-Distribution Preference Learningcited by 6
  24. 2024AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Datacited by 6
  25. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 6
  26. 2025SATER: A Self-Aware and Token-Efficient Approach to Routing and Cascadingcited by 6
  27. 2023Learning from Mistakes: Iterative Prompt Relabeling for Text-to-Image Diffusion Model Trainingcited by 5
  28. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Timecited by 5
  29. 2023An Interactive Framework for Profiling News Media Sourcescited by 4
  30. 2025Evaluating Language Models' Evaluations of Gamescited by 4
  31. 2026Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrationscited by 4
  32. 2024COrAL: Order-Agnostic Language Modeling for Efficient Iterative Refinementcited by 3
  33. 2025Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventionscited by 3
  34. 2025Learning Instruction-Following Policies through Open-Ended Instruction Relabeling with Large Language Modelscited by 2
  35. 2025TRPrompt: Bootstrapping Query-Aware Prompt Optimization from Textual Rewardscited by 2
  36. 2024On the Transformations across Reward Model, Parameter Update, and In-Context Promptcited by 1
  37. 2026Weird Generalization is Weirdly Brittlecited by 1
  38. 2026Delineating Knowledge Boundaries for Honest Large Vision-Language Modelscited by 1

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following
  2. 2023Chain of Hindsight Aligns Language Models with Feedback
  3. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
  4. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models
  5. 2024TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning
  6. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
  7. 2025REFINE-AF: A Task-Agnostic Framework to Align Language Models via Self-Generated Instructions using Reinforcement Learning from Automated Feedback
  8. 2026From Outcomes to Actions: Leveraging Hindsight for Long-Horizon Language Agent Training