mlsanityData to 14 Oct 2026

Feedback Loops With Language Models Drive In-Context Reward Hacking

arXiv 2402.06627 ↗First posted 9 Feb 2024cs.LG cs.AI cs.CLcited by 77

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Language models influence the external world: they query APIs that read and write to web pages, generate content that shapes human behavior, and run system commands as autonomous agents. These interactions form feedback loops: LLM outputs affect the world, which in turn affect subsequent LLM outputs. In this work, we show that feedback loops can cause in-context reward hacking (ICRH), where the LLM at test-time optimizes a (potentially implicit) objective but creates negative side effects in the process. For example, consider an LLM agent deployed to increase Twitter engagement; the LLM may retrieve its previous tweets into the context window and make them more controversial, increasing engagement but also toxicity. We identify and study two processes that lead to ICRH: output-refinement and policy-refinement. For these processes, evaluations on static datasets are insufficient -- they miss the feedback effects and thus cannot capture the most harmful behavior. In response, we provide three recommendations for evaluation to capture more instances of ICRH. As AI development accelerates, the effects of feedback loops will proliferate, increasing the need to understand their role in shaping LLM behavior.

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 60 of its 134 references that are in the map. First posted on arXiv 9 Feb 2024.

Builds on

60 of 134 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models10 Jan

Other references · 59

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  10. 2020Learning to summarize from human feedbackcited by 2,588
  11. 2020Extracting Training Data from Large Language Modelscited by 2,189
  12. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
38 more
  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  3. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  5. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  6. 2022Red Teaming Language Models with Language Modelscited by 1,026
  7. 2023Teaching Large Language Models to Self-Debugcited by 988
  8. 2021Ethical and social risks of harm from Language Modelscited by 966
  9. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  10. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  11. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  12. 2023Large Language Models as Optimizerscited by 864
  13. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  14. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  15. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  16. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  17. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  18. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  19. 2023Active Retrieval Augmented Generationcited by 516
  20. 2023Augmented Language Models: a Surveycited by 403
  21. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 364
  22. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  23. 2020Performative Predictioncited by 327
  24. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 301
  25. 2021Unsolved Problems in ML Safetycited by 292
  26. 2015Strategic Classificationcited by 257
  27. 2020Bias and Debias in Recommender System: A Survey and Future Directionscited by 240
  28. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 185
  29. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
  30. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  31. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 148
  32. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 146
  33. 2017How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utilitycited by 120
  34. 2020Feedback Loop and Bias Amplification in Recommender Systemscited by 90
  35. 2022PEER: A Collaborative Language Modelcited by 86
  36. 2024Escalation Risks from Language Models in Military and Diplomatic Decision-Makingcited by 63
  37. 2022Data Feedback Loops: Model-driven Amplification of Dataset Biasescited by 60
  38. 2023Can LLMs Follow Simple Rules?cited by 46

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 29; 2026: 37.

77 papers.

Sort:

By area:

  1. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  2. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
  3. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  4. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  5. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 169
  6. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  7. 2025ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Casescited by 65
  8. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 61
  9. 2025Inference-Time Reward Hacking in Large Language Modelscited by 34
  10. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  11. 2024Adversaries Can Misuse Combinations of Safe Modelscited by 32
  12. 2026ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent Trainingcited by 32
38 more
  1. 2024Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalizationcited by 23
  2. 2025Entropy-Gated Branching for Efficient Test-Time Reasoningcited by 22
  3. 2025A Survey on LLM Test-Time Compute via Search: Tasks, LLM Profiling, Search Algorithms, and Relevant Frameworkscited by 21
  4. 2025Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Modelscited by 18
  5. 2025Text2Grad: Reinforcement Learning from Natural Language Feedbackcited by 18
  6. 2026Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysiscited by 18
  7. 2025Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratchcited by 17
  8. 2024BELLS: A Framework Towards Future Proof Benchmarks for the Evaluation of LLM Safeguardscited by 14
  9. 2025Thinking beyond the anthropomorphic paradigm benefits LLM researchcited by 14
  10. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  11. 2024A Multi-AI Agent System for Autonomous Optimization of Agentic AI Solutions via Iterative Refinement and LLM-Driven Feedback Loopscited by 13
  12. 2024Accounting for AI and Users Shaping One Another: The Role of Mathematical Modelscited by 11
  13. 2025The Lock-in Hypothesis: Stagnation by Algorithmcited by 11
  14. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimizationcited by 10
  15. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  16. 2026Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Referencecited by 7
  17. 2025Towards Evaluating Proactive Risk Awareness of Multimodal Language Modelscited by 6
  18. 2025Specification Self-Correction: Mitigating In-Context Reward Hacking Through Test-Time Refinementcited by 6
  19. 2025TDFlow: Agentic Workflows for Test Driven Developmentcited by 6
  20. 2024Impact of Decentralized Learning on Player Utilities in Stackelberg Gamescited by 5
  21. 2025Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correctioncited by 5
  22. 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system securitycited by 5
  23. 2025Psychometric Personality Shaping Modulates Capabilities and Safety in Language Modelscited by 4
  24. 2026Bridging Online and Offline RL: Contextual Bandit Learning for Multi-Turn Code Generationcited by 4
  25. 2025Large Language Models for Bioinformaticscited by 3
  26. 2025Enhancing Conversational Agents with Theory of Mind: Aligning Beliefs, Desires, and Intentions for Human-Like Interactioncited by 3
  27. 2025Giving AI Agents Access to Cryptocurrency and Smart Contracts Creates New Vectors of AI Harmcited by 3
  28. 2025A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challengescited by 3
  29. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Modelscited by 3
  30. 2026The Compliance Gap: Why AI Systems Promise to Follow Process Instructions but Don'tcited by 3
  31. 2026LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrailscited by 3
  32. 2025Convergence Dynamics and Stabilization Strategies of Co-Evolving Generative Modelscited by 2
  33. 2025Understanding the Impact of Sampling Quality in Direct Preference Optimizationcited by 2
  34. 2025Causally-Enhanced Reinforcement Policy Optimizationcited by 2
  35. 2026Distilling Feedback into Memory-as-a-Toolcited by 2
  36. 2026Self-Trained Verification for Training- and Test-Time Self-Improvementcited by 2
  37. 2026Verify, Repair, Repeat, or Stop? Robust Stopping for Noisy Verify-Repair Loops in LLM Agentscited by 2
  38. 2026Emergent Collusion in Long-Horizon LLM Agent Interactioncited by 2

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Spontaneous Reward Hacking in Iterative Self-Refinement
  2. 2026Large Language Models Hack Rewards, and Society
  3. 2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
  4. 2026Do LLMs Learn from Rewards in Context? : Rethinking the role of reward in In-Context Reinforcement Learning
  5. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs
  6. 2026Pressure, Context, and Machine Self-Control: A Criminological Test of Reward Hacking in Generative AI Models
  7. 2026From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agents
  8. 2026Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds