mlsanityData to 7 Oct 2026

Spontaneous Reward Hacking in Iterative Self-Refinement

arXiv 2407.04549 ↗First posted 5 Jul 2024cs.CL cs.AIcited by 24

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Language models are capable of iteratively improving their outputs based on natural language feedback, thus enabling in-context optimization of user preference. In place of human users, a second language model can be used as an evaluator, providing feedback along with numerical ratings which the generator attempts to optimize. However, because the evaluator is an imperfect proxy of user preference, this optimization can lead to reward hacking, where the evaluator's ratings improve while the generation quality remains stagnant or even decreases as judged by actual user preference. The concern of reward hacking is heightened in iterative self-refinement where the generator and the evaluator use the same underlying language model, in which case the optimization pressure can drive them to exploit shared vulnerabilities. Using an essay editing task, we show that iterative self-refinement leads to deviation between the language model evaluator and human judgment, demonstrating that reward hacking can occur spontaneously in-context with the use of iterative self-refinement. In addition, we study conditions under which reward hacking occurs and observe two factors that affect reward hacking severity: model size and context sharing between the generator and the evaluator.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 27 references that are in the map. First posted on arXiv 5 Jul 2024.

Builds on

20 of 27 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2016Concrete Problems in AI Safetycited by 1,582
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  7. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  8. 2023Teaching Large Language Models to Self-Debugcited by 963
  9. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  11. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  12. 2022Self-critiquing models for assisting human evaluatorscited by 372
8 more
  1. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 246
  2. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  3. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  4. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  5. 2023Can Large Language Models Really Improve by Self-critiquing Their Own Plans?cited by 107
  6. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100
  7. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  8. 2023"Oops, Did I Just Say That?" Testing and Repairing Unethical Suggestions of Large Language Models with Suggest-Critique-Reflect Processcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 12.

24 papers.

Sort:

By area:

  1. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  2. 2024Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Editscited by 49
  3. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 38
  4. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  5. 2025AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computationcited by 24
  6. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  7. 2024Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinementcited by 13
  8. 2024Honesty to Subterfuge: In-Context Reinforcement Learning Can Make Honest Models Reward Hackcited by 8
  9. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  10. 2026More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judgescited by 6
  11. 2024SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Marginscited by 5
  12. 2026Stop Hand-Holding Your Coding Agent: Engineering the Loops that Replace Step-by-Step Promptingcited by 5
12 more
  1. 2026Bridging Online and Offline RL: Contextual Bandit Learning for Multi-Turn Code Generationcited by 4
  2. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  3. 2026Memory Reward Inflation in Self-Improving LLM Agentscited by 3
  4. 2025L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Trainingcited by 2
  5. 2025Causally-Enhanced Reinforcement Policy Optimizationcited by 2
  6. 2026Verify, Repair, Repeat, or Stop? Robust Stopping for Noisy Verify-Repair Loops in LLM Agentscited by 2
  7. 2026LLM-as-a-Judge Scores Are Unreliable Optimization Signals in Closed-Loop Table Recognitioncited by 1
  8. 2026LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrailscited by 1
  9. 2026Beyond Uncertainty: Multi-Solver Disagreement Rewards for Self-Evolving Reasoning Curriculacited by 0
  10. 2026Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Promptscited by 0
  11. 2026Scope Before You Persist: Preventing Cross-Family Interference in Agent Memorycited by 0
  12. 2026Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-Generated Critiques Boost Reward Modeling for Language Models
  2. 2024Feedback Loops With Language Models Drive In-Context Reward Hacking
  3. 2025Does Self-Evaluation Enable Wireheading in Language Models?
  4. 2026Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
  5. 2026Harness-agnostic detection and immunization of reward hacking in self-evolving language models
  6. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs
  7. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges
  8. 2026Pressure, Context, and Machine Self-Control: A Criminological Test of Reward Hacking in Generative AI Models