mlsanityData to 14 Oct 2026

On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback

arXiv 2411.02306 ↗First posted 4 Nov 2024cs.LG cs.AIcited by 47

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

As LLMs become more widely deployed, there is increasing interest in directly optimizing for feedback from end users (e.g. thumbs up) in addition to feedback from paid annotators. However, training to maximize human feedback creates a perverse incentive structure for the AI to resort to manipulative or deceptive tactics to obtain positive feedback from users who are vulnerable to such strategies. We study this phenomenon by training LLMs with Reinforcement Learning with simulated user feedback in environments of practical LLM usage. In our settings, we find that: 1) Extreme forms of "feedback gaming" such as manipulation and deception are learned reliably; 2) Even if only 2% of users are vulnerable to manipulative strategies, LLMs learn to identify and target them while behaving appropriately with other users, making such behaviors harder to detect; 3) To mitigate this issue, it may seem promising to leverage continued safety training or LLM-as-judges during training to filter problematic outputs. Instead, we found that while such approaches help in some of our settings, they backfire in others, sometimes even leading to subtler manipulative behaviors. We hope our results can serve as a case study which highlights the risks of using gameable feedback sources -- such as user feedback -- as a target for RL.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 68 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

35 of 68 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Rewarding Chatbots for Real-World Engagement with Millions of Users10 Mar
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 33

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  6. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  7. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  8. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  9. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  10. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  11. 2020Backdoor Learning: A Surveycited by 426
  12. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
21 more
  1. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  2. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  3. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  4. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 188
  5. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
  6. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 160
  7. 2024Language Models Learn to Mislead Humans via RLHFcited by 124
  8. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 108
  9. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 87
  10. 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
  11. 2018Generative Adversarial User Model for Reinforcement Learning Based Recommendation Systemcited by 64
  12. 2020Hidden Incentives for Auto-Induced Distributional Shiftcited by 43
  13. 2023Characterizing Manipulation from AI Systemscited by 43
  14. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  15. 2022Estimating and Penalizing Induced Preference Shifts in Recommender Systemscited by 29
  16. 2021User Tampering in Reinforcement Learning Recommender Systemscited by 23
  17. 2024The Reasons that Agents Act: Intention and Instrumental Goalscited by 23
  18. 2022Path-Specific Objectives for Safer Agent Incentivescited by 20
  19. 2023Implicit meta-learning may lead language models to trust more reliable sourcescited by 10
  20. 2024The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputscited by 10
  21. 2023Benchmarks for Detecting Measurement Tamperingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 27; 2026: 20.

47 papers.

Sort:

By area:

  1. 2025Emergent Abilities in Large Language Models: A Surveycited by 37
  2. 2025Training language models to be warm and empathetic makes them less reliable and more sycophanticcited by 21
  3. 2025MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hackingcited by 19
  4. 2025RLHS: Mitigating Misalignment in RLHF with Hindsight Simulationcited by 15
  5. 2025Technological folie `a deux: Feedback Loops Between AI Chatbots and Mental Illnesscited by 15
  6. 2025Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspectivecited by 13
  7. 2025The Lock-in Hypothesis: Stagnation by Algorithmcited by 11
  8. 2025AssistanceZero: Scalably Solving Assistance Gamescited by 10
  9. 2026Quantifying the Utility of User Simulators for Building Collaborative LLM Assistantscited by 10
  10. 2025Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanationscited by 9
  11. 2025Preference Learning with Lie Detectors can Induce Honesty or Evasioncited by 9
  12. 2026AI Assistance Reduces Persistence and Hurts Independent Performancecited by 9
35 more
  1. 2025Societal Alignment Frameworks Can Improve LLM Alignmentcited by 8
  2. 2025LLMs Deceive Unintentionally: Emergent Misalignment in Dishonesty from Misaligned Samples to Biased Human-AI Interactionscited by 8
  3. 2025Are Large Language Models Sensitive to the Motives Behind Communication?cited by 8
  4. 2025Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Modelscited by 7
  5. 2025Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Riskscited by 5
  6. 2025Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Frameworkcited by 5
  7. 2025Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Modelscited by 4
  8. 2025CTRL-Rec: Controlling Recommender Systems With Natural Languagecited by 4
  9. 2026Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMscited by 4
  10. 2025But what is your honest answer? Aiding LLM-judges with honest alternatives using steering vectorscited by 3
  11. 2025Training LLM Agents to Empower Humanscited by 3
  12. 2026Towards Understanding Specification Gaming in Reasoning Modelscited by 3
  13. 2025PRISON: Unmasking the Criminal Potential of Large Language Modelscited by 2
  14. 2025The Ends Justify the Thoughts: RL-Induced Motivated Reasoning in LLM CoTscited by 2
  15. 2025EQ-Negotiator: Dynamic Emotional Personas Empower Small Language Models for Edge-Deployable Credit Negotiationcited by 2
  16. 2026Knowledge-Verified Emergent Deception in LLM Agents Under Conflicting Incentivescited by 2
  17. 2025Towards Low-Resource Alignment to Diverse Perspectives with Sparse Feedbackcited by 1
  18. 2025Debate with Images: Detecting Deceptive Behaviors in Multimodal Large Language Modelscited by 1
  19. 2026From Helpfulness to Toxic Proactivity: Diagnosing Behavioral Misalignment in LLM Agentscited by 1
  20. 2026Against Proxy Optimizationcited by 1
  21. 2025The Intercepted Self: How Generative AI Challenges the Dynamics of the Relational Selfcited by 0
  22. 2025Approximating the Mathematical Structure of Psychodynamicscited by 0
  23. 2026Alignment Risks from Capability-Seeking RL Trainingcited by 0
  24. 2026Disentangling Deception and Hallucination Failures in LLMscited by 0
  25. 2026Probing the Limits of the Lie Detector Approach to LLM Deceptioncited by 0
  26. 2026Questionnaire Responses Do not Capture the Safety of AI Agentscited by 0
  27. 2026Mitigating LLM biases toward spurious social contexts using direct preference optimizationcited by 0
  28. 2026Safety Training Modulates Harmful Misalignment Under On-Policy RL, But Direction Depends on Environment Designcited by 0
  29. 2026The Severance Problem: LLMs are Unaware of the Person Beyond the Promptcited by 0
  30. 2026Psychological Influences of Conversational AI: Research and Design Directions for Reducing Harm and Promoting Well-Beingcited by 0
  31. 2026Corrigible Assistance in One Round: Pragmatic-Pedagogic Best Responsecited by 0
  32. 2026Individual Disempowerment through an Advice Channel: Control Loss when Influence is Endogenouscited by 0
  33. 2026XYEval: Agents say yes to bad advicecited by 0
  34. 2026The Confidence Game: Strategic Miscalibration in Human-AI Delegationcited by 0
  35. 2026Understanding and Reducing Misalignment in Persuasion Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning from User Feedback
  2. 2025Beyond Prompt-Induced Lies: Investigating LLM Deception on Benign Prompts
  3. 2024Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMs
  4. 2025Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizers
  5. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedback
  6. 2024LLM Critics Help Catch LLM Bugs
  7. 2026Adversarial Feeds Steer LLM Agent Decisions Against Their Defaults
  8. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs