mlsanityData to 14 Oct 2026

Linear Probe Penalties Reduce LLM Sycophancy

arXiv 2412.00967 ↗First posted 1 Dec 2024cs.AIcited by 28

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Large language models (LLMs) are often sycophantic, prioritizing agreement with their users over accurate or objective statements. This problematic behavior becomes more pronounced during reinforcement learning from human feedback (RLHF), an LLM fine-tuning stage intended to align model outputs with human values. Instead of increasing accuracy and reliability, the reward model learned from RLHF often rewards sycophancy. We develop a linear probing method to identify and penalize markers of sycophancy within the reward model, producing rewards that discourage sycophantic behavior. Our experiments show that constructing and optimizing against this surrogate reward function reduces sycophantic behavior in multiple open-source LLMs. Our results suggest a generalizable methodology for reducing unwanted LLM behaviors that are not sufficiently disincentivized by RLHF fine-tuning.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 25 references that are in the map. First posted on arXiv 1 Dec 2024.

Builds on

13 of 25 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 11

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  4. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  6. 2023Large Language Modelscited by 780
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  8. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  9. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
  10. 2023Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblockscited by 78
  11. 2014Human language reveals a universal positivity biascited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 15.

28 papers.

Sort:

By area:

  1. 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 129
  2. 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 104
  3. 2025Sycophancy Is Not One Thing: Causal Separation of Sycophantic Behaviors in LLMscited by 37
  4. 2025BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMscited by 27
  5. 2025RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?cited by 24
  6. 2026Sycophancy Hides Linearly in the Attention Headscited by 23
  7. 2026How RLHF Amplifies Sycophancycited by 23
  8. 2025Sycophancy under Pressure: Evaluating and Mitigating Sycophantic Bias via Adversarial Dialogues in Scientific QAcited by 18
  9. 2025Beyond Token Probes: Hallucination Detection via Activation Tensors with ACT-ViTcited by 17
  10. 2025Consistency Training Helps Stop Sycophancy and Jailbreakscited by 16
  11. 2026What Counts as AI Sycophancy? A Taxonomy and Expert Survey of a Fragmented Constructcited by 16
  12. 2026The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probescited by 12
16 more
  1. 2025Preference Learning with Lie Detectors can Induce Honesty or Evasioncited by 9
  2. 2025Pointing to a Llama and Call it a Camel: On the Sycophancy of Multimodal Large Language Modelscited by 5
  3. 2025LPASS: Linear Probes as Stepping Stones for vulnerability detection using compressed LLMscited by 4
  4. 2025PENDULUM: A Benchmark for Assessing Sycophancy in Multimodal Large Language Modelscited by 4
  5. 2026Automatically Finding Reward Model Biasescited by 4
  6. 2026Pressure, What Pressure? Sycophancy Disentanglement in Language Models via Reward Decompositioncited by 4
  7. 2026CausalT5k: Diagnosing Refusal and Failure Modes in Trustworthy Causal Reasoning Across Causal Rungscited by 2
  8. 2026Generative AI for Managerial Decision-Making under Ambiguity and Sycophancycited by 1
  9. 2026Reasoning Models Know What's Important, and Encode It in Their Activationscited by 1
  10. 2026Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generationcited by 1
  11. 2026Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gatingcited by 1
  12. 2026Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMscited by 1
  13. 2025SMARTER: A Data-efficient Framework to Improve Toxicity Detection with Explanation via Self-augmenting Large Language Modelscited by 0
  14. 2026Measuring and Detecting Harmful AI Sycophancycited by 0
  15. 2026Mitigating LLM sycophancy with RL-based fine-tuning: Bayesian Truth Serum approachcited by 0
  16. 2026Mitigating Social Sycophancy via Pluralistic Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Calibration Collapse Under Sycophancy Fine-Tuning: How Reward Hacking Breaks Uncertainty Quantification in LLMs
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  3. 2026Probe with Participation Trophies: Random-Reward RL as a Probe of LLM Capability
  4. 2024Sycophancy in Large Language Models: Causes and Mitigations
  5. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Design
  6. 2024Post-hoc Reward Calibration: A Case Study on Length Bias
  7. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
  8. 2026Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design