mlsanityData to 7 Oct 2026

Language Models Learn to Mislead Humans via RLHF

arXiv 2409.12822 ↗First posted 19 Sep 2024cs.CLcited by 118

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Language models (LMs) can produce errors that are hard to detect for humans, especially when the task is complex. RLHF, the most popular post-training method, may exacerbate this problem: to achieve higher rewards, LMs might get better at convincing humans that they are right even when they are wrong. We study this phenomenon under a standard RLHF pipeline, calling it "U-SOPHISTRY" since it is Unintended by model developers. Specifically, we ask time-constrained (e.g., 3-10 minutes) human subjects to evaluate the correctness of model outputs and calculate humans' accuracy against gold labels. On a question-answering task (QuALITY) and programming task (APPS), RLHF makes LMs better at convincing our subjects but not at completing the task correctly. RLHF also makes the model harder to evaluate: our subjects' false positive rate increases by 24.1% on QuALITY and 18.3% on APPS. Finally, we show that probing, a state-of-the-art approach for detecting Intended Sophistry (e.g. backdoored LMs), does not generalize to U-SOPHISTRY. Our results highlight an important failure mode of RLHF and call for more research in assisting humans to align them.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 35 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

25 of 35 references.

References · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  6. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  7. 2021Measuring Coding Challenge Competence With APPScited by 879
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  9. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  10. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  11. 2022Self-critiquing models for assisting human evaluatorscited by 372
  12. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
13 more
  1. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  2. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  3. 2023The False Promise of Imitating Proprietary LLMscited by 254
  4. 2023An Overview of Catastrophic AI Riskscited by 214
  5. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  6. 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
  7. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  8. 2024LLM Critics Help Catch LLM Bugscited by 138
  9. 2023Human Feedback is not Gold Standardcited by 82
  10. 2023Debate Helps Supervise Unreliable Expertscited by 39
  11. 2022Two-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension Questionscited by 16
  12. 2024Learning Task Decomposition to Assist Humans in Competitive Programmingcited by 14
  13. 2022Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQLcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 52; 2026: 53.

118 papers.

Sort:

By area:

  1. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  2. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  3. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  4. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 187
  5. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 105
  6. 2024Sycophancy in Large Language Models: Causes and Mitigationscited by 97
  7. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  8. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  9. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  10. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  11. 2025Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verificationcited by 49
  12. 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedbackcited by 45
38 more
  1. 2025Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoningcited by 29
  2. 2024Adaptive Deployment of Untrusted LLMs Reduces Distributed Threatscited by 27
  3. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMscited by 25
  4. 2025Mitigating Deceptive Alignment via Self-Monitoringcited by 25
  5. 2025Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluationcited by 23
  6. 2026Sycophancy Hides Linearly in the Attention Headscited by 21
  7. 2025MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hackingcited by 19
  8. 2025Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Modelscited by 18
  9. 2026Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphscited by 17
  10. 2024Training Language Models to Win Debates with Self-Play Improves Judge Accuracycited by 16
  11. 2024Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineeringcited by 16
  12. 2025RLHS: Mitigating Misalignment in RLHF with Hindsight Simulationcited by 15
  13. 2024Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Modelscited by 14
  14. 2025Technological folie `a deux: Feedback Loops Between AI Chatbots and Mental Illnesscited by 14
  15. 2024Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can't Answer?cited by 13
  16. 2025Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PDcited by 13
  17. 2025An alignment safety case sketch based on debatecited by 12
  18. 2025AI Safety Should Prioritize the Future of Workcited by 11
  19. 2025Unsupervised Elicitation of Language Modelscited by 11
  20. 2025Recontextualization Mitigates Specification Gaming without Modifying the Specificationcited by 11
  21. 2025Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanationscited by 9
  22. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 9
  23. 2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hackingcited by 9
  24. 2025Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approachcited by 8
  25. 2025PredictaBoard: Benchmarking LLM Score Predictabilitycited by 7
  26. 2025RedDebate: Safer Responses Through Multi-Agent Red Teaming Debatescited by 7
  27. 2025Multiplayer Nash Preference Optimizationcited by 7
  28. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  29. 2025Evaluating & Reducing Deceptive Dialogue From Language Models with Multi-turn RLcited by 7
  30. 2026Toward Expert Investment Teams:A Multi-Agent LLM System with Fine-Grained Trading Taskscited by 7
  31. 2025AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Modelscited by 6
  32. 2025Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Modelscited by 6
  33. 2025A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Userscited by 6
  34. 2025The Impact of Off-Policy Training Data on Probe Generalisationcited by 6
  35. 2026QuRL: Efficient Reinforcement Learning with Quantized Rolloutcited by 6
  36. 2026More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judgescited by 6
  37. 2024Mitigating Sycophancy in Decoder-Only Transformer Architectures: Synthetic Data Interventioncited by 5
  38. 2025Iterative Label Refinement Matters More than Preference Optimization under Weak Supervisioncited by 5

50 most cited of 118 All 118 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Teaching language models to support answers with verified quotes
  2. 2024The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models
  3. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
  4. 2025Resisting Correction: How RLHF Makes Language Models Ignore External Safety Signals in Natural Conversation
  5. 2023Large Language Model Unlearning
  6. 2025Large Language Models are overconfident and amplify human bias
  7. 2025Rectifying Belief Space via Unlearning to Harness LLMs' Reasoning
  8. 2023Aligning Neural Machine Translation Models: Human Feedback in Training and Inference