# Pressure, What Pressure? Sycophancy Disentanglement in Language Models via Reward Decomposition

arXiv: https://arxiv.org/abs/2604.05279 · HTML: https://mlsanity.com/paper/2604.05279/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 7 Apr 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 20 of 20)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Simple synthetic data reduces sycophancy in large language models](https://mlsanity.com/paper/2308.03958/) (2023)
- [SycEval: Evaluating LLM Sycophancy](https://mlsanity.com/paper/2502.08177/) (2025)
- [Sycophancy in Large Language Models: Causes and Mitigations](https://mlsanity.com/paper/2411.15287/) (2024)
- [When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviour](https://mlsanity.com/paper/2311.09410/) (2023)
- [From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning](https://mlsanity.com/paper/2409.01658/) (2024)
- [Sycophancy Is Not One Thing: Causal Separation of Sycophantic Behaviors in LLMs](https://mlsanity.com/paper/2509.21305/) (2025)
- [Be Friendly, Not Friends: How LLM Sycophancy Shapes User Trust](https://mlsanity.com/paper/2502.10844/) (2025)
- [Linear Probe Penalties Reduce LLM Sycophancy](https://mlsanity.com/paper/2412.00967/) (2024)
- [How RLHF Amplifies Sycophancy](https://mlsanity.com/paper/2602.01002/) (2026)
- [Reasoning Isn't Enough: Examining Truth-Bias and Sycophancy in LLMs](https://mlsanity.com/paper/2506.21561/) (2025)

## Built on by (most cited first, 4 of 4)

- [Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs](https://mlsanity.com/paper/2606.16011/) (2026)
- [Resist and Update: Counterfactual Report Coordinates for Incentive-Compatible LLMs](https://mlsanity.com/paper/2607.12985/) (2026)
- [Resist, Update, Reject: Preference Optimization Installs a Prior-Dependent Reliability Switch](https://mlsanity.com/paper/2609.22359/) (2026)
- [RelLedger: Relational Information Governance for LLM Agent Organizations](https://mlsanity.com/paper/2610.16281/) (2026)

## Abstract (arXiv)

Large language models exhibit sycophancy, the tendency to shift their stated positions toward perceived user preferences or authority cues regardless of evidence. Standard alignment methods fail to correct this because scalar reward models conflate two distinct failure modes into a single signal: pressure capitulation, where the model changes a correct answer under social pressure, and evidence blindness, where the model ignores the provided context entirely. We operationalise sycophancy through formal definitions of pressure independence and evidence responsiveness, serving as a working framework for disentangled training rather than a definitive characterisation of the phenomenon. We propose the first approach to sycophancy reduction via reward decomposition, introducing a multi-component Group Relative Policy Optimisation (GRPO) reward that decomposes the training signal into five terms: pressure resistance, context fidelity, position consistency, agreement suppression, and factual correctness. We train using a contrastive dataset pairing pressure-free baselines with pressured variants across three authority levels and two opposing evidence contexts. Across five base models, our two-phase pipeline consistently reduces sycophancy on all metric axes, with ablations confirming that each reward term governs an independent behavioural dimension. The learned resistance to pressure generalises beyond our training methodology and prompt structure, reducing answer-priming sycophancy by up to 17 points on SycophancyEval despite the absence of such pressure forms during training.