mlsanityData to 14 Oct 2026

Learning When to Trust in Contextual Social Bandits

arXiv 2603.13356 ↗First posted 9 Mar 2026cs.AI cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Robust reinforcement learning typically assumes that feedback sources are either globally trustworthy or corrupted within a fixed global budget. We identify a more subtle failure mode that escapes this dichotomy, which we call Contextual Sycophancy. In this failure, evaluators are truthful in benign contexts but systematically biased in critical ones, so that no single evaluator is reliable everywhere and the corrupt evaluators may form a majority in the contexts that matter. Our first result is an information-theoretic lower bound. We exhibit two problem instances that induce identical social-feedback distributions yet have disjoint optimal actions, proving that any algorithm relying on social feedback alone (including any robust aggregator, regardless of breakdown point) incurs Omega(T) latent regret. This shows that breaking contextual sycophancy is impossible without having some information. We then show that a sparse stream of ground-truth audits, available with probability paud, is sufficient. We propose ESA, which learns a per-evaluator contextual trust boundary from audits and re-weights feedback accordingly, and we prove a high-probability latent-regret bound of tildeO!big(√T,dVC/paud + d√T + εtolTbig), where dVC is the complexity of the adversary's bias strategy. The audit-dependence 1/√paud matches the information-theoretic necessity of audits. Empirically, ESA recovers the ground truth when 80% of the social layer is adversarial, a regime in which median- and mean-based robust baselines fail.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 24 of its 40 references that are in the map. First posted on arXiv 9 Mar 2026.

Builds on

24 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026Objective Decoupling in Social Reinforcement Learning: Recovering Ground Truth from Sycophantic Majorities8 Feb

Other references · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  4. 2016Concrete Problems in AI Safetycited by 1,642
  5. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  6. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  7. 2017Robust Adversarial Reinforcement Learningcited by 451
  8. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  9. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
  10. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  11. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
  12. 2020Neural Thompson Samplingcited by 128
11 more
  1. 2018Stochastic bandits robust to adversarial corruptionscited by 119
  2. 2019Better Algorithms for Stochastic Bandits with Adversarial Corruptionscited by 92
  3. 2020Stochastic Linear Bandits Robust to Adversarial Attackscited by 65
  4. 2024From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuningcited by 54
  5. 2019Improving Neural Language Modeling via Adversarial Trainingcited by 49
  6. 2015A Survey on Truth Discoverycited by 36
  7. 2021Robust Stochastic Linear Contextual Bandits Under Adversarial Attackscited by 33
  8. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitationscited by 11
  9. 2024Towards Understanding the Robustness of LLM-based Evaluations under Perturbationscited by 10
  10. 2023Fairness and Bias in Truth Discovery Algorithms: An Experimental Analysiscited by 4
  11. 2025Impact of Label Noise from Large Language Models Generated Annotations on Evaluation of Diagnostic Model Performancecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Corruption-Robust Linear Bandits: Minimax Optimality and Gap-Dependent Misspecification
  2. 2026Interpreting Reinforcement Learning Agents with Susceptibilities
  3. 2024Corruption Robust Offline Reinforcement Learning with Human Feedback
  4. 2023Adversarial Rewards in Universal Learning for Contextual Bandits
  5. 2024Learning from Imperfect Human Feedback: a Tale from Corruption-Robust Dueling
  6. 2024Nearly Optimal Algorithms for Contextual Dueling Bandits from Adversarial Feedback
  7. 2025Sparse Offline Reinforcement Learning with Corruption Robustness
  8. 2026Learning to Trust Experience: A Monitor-Trust-Regulator Framework for Learning under Unobservable Feedback Reliability