# Objective Decoupling in Social Reinforcement Learning: Recovering Ground Truth from Sycophantic Majorities

arXiv: https://arxiv.org/abs/2602.08092 · HTML: https://mlsanity.com/paper/2602.08092/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 8 Feb 2026.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 15 of 15)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Reinforcement Learning with Perturbed Rewards](https://mlsanity.com/paper/1810.01032/) (2018)
- [RRM: Robust Reward Model Training Mitigates Reward Hacking](https://mlsanity.com/paper/2409.13156/) (2024)
- [A Survey on Truth Discovery](https://mlsanity.com/paper/1505.02463/) (2015)
- [Robust LLM Alignment via Distributionally Robust Direct Preference Optimization](https://mlsanity.com/paper/2502.01930/) (2025)
- [Three Dogmas of Reinforcement Learning](https://mlsanity.com/paper/2407.10583/) (2024)
- [Sign-SZPO: Provable Preference-based Reinforcement Learning with an Unknown Link Function](https://mlsanity.com/paper/2506.03066/) (2025)
- [One Goal, Many Challenges: Robust Preference Optimization Amid Content-Aware and Multi-Source Noise](https://mlsanity.com/paper/2503.12301/) (2025)

## Built on by (most cited first, 2 of 2)

- [Learning When to Trust in Contextual Social Bandits](https://mlsanity.com/paper/2603.13356/) (2026)
- [Training and Evaluating Ethical Reinforcement Learning Agents on Per-Episode Distributions](https://mlsanity.com/paper/2608.14642/) (2026)

## Abstract (arXiv)

Contemporary AI alignment strategies rely on a fragile premise: that human feedback, while noisy, remains a fundamentally truthful signal. In this paper, we identify this assumption as Dogma 4 of Reinforcement Learning (RL). We demonstrate that while this dogma holds in static environments, it fails in social settings where evaluators may be sycophantic, lazy, or adversarial. We prove that under Dogma 4, standard RL agents suffer from what we call Objective Decoupling, a structural failure mode where the agent's learned objective permanently separates from the latent ground truth, guaranteeing convergence to misalignment. To resolve this, we propose Epistemic Source Alignment (ESA). Unlike standard robust methods that rely on statistical consensus (trusting the majority), ESA utilizes sparse safety axioms to judge the source of the feedback rather than the signal itself. We prove that this "judging the judges" mechanism guarantees convergence to the true objective, even when a majority of evaluators are biased. Empirically, we show that while traditional consensus methods fail under majority collusion, our approach successfully recovers the optimal policy.