mlsanityData to 14 Oct 2026

Explaining and Preventing Alignment Collapse in Iterative RLHF

arXiv 2605.04266 ↗First posted 5 May 2026cs.LG stat.MLcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) typically assumes a static or non-strategic reward model (RM). In iterative deployment, however, the policy generates the data on which the RM is retrained, creating a feedback loop. Building on the Stackelberg game formulation of this interaction, we derive an analytical decomposition of the policy's true optimization gradient into a standard policy gradient and a parameter-steering term that captures the policy's influence on the RM's future parameters. We show that standard iterative RLHF, which drops this steering term entirely, suffers from alignment collapse: the policy systematically exploits the RM's blind spots, producing low-quality, high-reward outputs whose feedback reinforces the very errors it exploits. To mitigate this, we propose foresighted policy optimization (FPO), a mechanism-design intervention that restores the missing steering term by regularizing the policy's parameter-steering effect on RM updates. We instantiate FPO via a scalable first-order approximation and demonstrate that it prevents alignment collapse on both controlled environments and an LLM alignment pipeline using Llama-3.2-1B.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 62 of its 80 references that are in the map. First posted on arXiv 5 May 2026.

Builds on

62 of 80 references.

Closest ancestors · 25

Same direction, newest first

  1. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards20 Feb
  2. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective31 Jan
  3. 2025Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game18 Dec
  4. 2025Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback21 Jul
  5. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspective19 Mar
  6. 2025Reward Shaping to Mitigate Reward Hacking in RLHF26 Feb
  7. 2025Understanding Impact of Human Feedback via Influence Functions10 Jan
  8. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms5 Jun
  9. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  10. 2024Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF10 Feb
  11. 2024Self-Rewarding Language Models18 Jan
  12. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
13 more
  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  2. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  3. 2023Nash Learning from Human Feedback1 Dec
  4. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  5. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  6. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  7. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  8. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback3 Aug
  9. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  11. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  12. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  13. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
25 more
  1. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  6. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  7. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  8. 2019Meta-Learning with Implicit Gradientscited by 525
  9. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  10. 2020Performative Predictioncited by 327
  11. 2016Hyperparameter optimization with approximate gradientcited by 233
  12. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  13. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 101
  14. 2021Outside the Echo Chamber: Optimizing the Performative Riskcited by 96
  15. 2024Asymptotics of Language Model Alignmentcited by 47
  16. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Modelscited by 20
  17. 2021Stochastic Optimization under Distributional Driftcited by 15
  18. 2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignmentcited by 10
  19. 2025Statistical Collusion by Collectives on Learning Platformscited by 6
  20. 2025Performative Policy Gradient: Optimality in Performative Reinforcement Learningcited by 6
  21. 2026The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safetycited by 4
  22. 2025Bi-Level Policy Optimization with Nystr"om Hypergradientscited by 3
  23. 2026Anytime Detection of Strategic Deviations in Multi-Agent Systemscited by 2
  24. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamicscited by 2
  25. 2026Mitigating Reward Hacking in RLHF via Advantage Sign Robustnesscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Multimodal Reward Hacking in Reinforcement Learningcited by 2
  2. 2026Efficient Hypergradient Descent for Inverse Reinforcement Learningcited by 0
  3. 2026Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Controlcited by 0
  4. 2026Self-Consuming Generative Models with Co-Evolving Human Preferencescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Unifying Stable Optimization and Reference Regularization in RLHF
  2. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  3. 2026SAFE: Stable Alignment Finetuning with Entropy-Aware Predictive Control for Reinforcement Learning from Human Feedback (RLHF)
  4. 2024Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment
  5. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
  6. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization
  7. 2025RLHS: Mitigating Misalignment in RLHF with Hindsight Simulation
  8. 2025Reward Model Overoptimisation in Iterated RLHF