mlsanityData to 14 Oct 2026

Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectories

arXiv 2509.16742 ↗First posted 20 Sep 2025cs.AIcited by 11

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Despite the remarkable capabilities of large language models, current training paradigms inadvertently foster textitsycophancy, i.e., the tendency of a model to agree with or reinforce user-provided information even when it's factually incorrect. To address this challenge, we introduce SMART (Sycophancy Mitigation through Adaptive Reasoning Trajectories), which reframes sycophancy as a textitreasoning optimization problem rather than an output alignment issue. SMART is a two-stage framework comprising: (1) Uncertainty-Aware Adaptive Monte Carlo Tree Search (UA-MCTS), which dynamically adjusts model exploration based on state-level uncertainty to collect high-quality, diverse reasoning trajectories alongside both stepwise progress and final outcome rewards; and (2) progress-based reinforcement learning, which fine-tunes the model using the collected trajectories and reward signals to reinforce effective reasoning patterns. Through extensive experiments, we show that SMART significantly reduces sycophantic behavior while preserving strong performance on out-of-distribution inputs and maintaining general capabilities. These results underscore the importance of optimizing internal reasoning mechanisms to build more truthful and aligned AI assistants.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 34 of its 50 references that are in the map. First posted on arXiv 20 Sep 2025.

Builds on

34 of 50 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modeling7 Mar
  2. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning6 Mar
  3. 2025Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Search6 Mar
  4. 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May

Other references · 29

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2023Mistral 7Bcited by 3,164
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  10. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  11. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  12. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
17 more
  1. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  3. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  4. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  5. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  6. 2022Self-critiquing models for assisting human evaluatorscited by 380
  7. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  8. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 279
  9. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
  10. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
  11. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
  12. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  13. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 94
  14. 2024From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuningcited by 54
  15. 2024Sycophancy in Vision-Language Models: A Systematic Analysis and an Inference-Time Mitigation Frameworkcited by 17
  16. 2024Have the VLMs Lost Confidence? A Study of Sycophancy in VLMscited by 15
  17. 2024Chaos with Keywords: Exposing Large Language Models Sycophantic Hallucination to Misleading Keywords and Evaluating Defense Strategiescited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026What Counts as AI Sycophancy? A Taxonomy and Expert Survey of a Fragmented Constructcited by 16
  2. 2026MemSyco-Bench: Benchmarking Sycophancy in Agent Memorycited by 12
  3. 2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hackingcited by 11
  4. 2026A Mechanistic View of Authority Hierarchy in LLM Sycophancycited by 9
  5. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  6. 2026Too Polite to Disagree: Understanding Sycophancy Propagation in Multi-Agent Systemscited by 6
  7. 2026Proxy Reward Internalization and Mechanistic Exploitation: A Learned Precursor to Reward Hacking and Its Generalizationcited by 1
  8. 2026Caved or Convinced: Temporal Sampling Gates Claim Deference in Video Large Language Modelscited by 1
  9. 2026LLMs Trust Their Own: Identity-Dependent Conformity in Multi-Agent Systemscited by 1
  10. 2026Agents Don't Just Agree, They Remember: Benchmarking Persistent Sycophancy in Self-Improving Personal Agentscited by 0
  11. 2026Suppressing Pressure, Amplifying Evidence: Self-Guided Attention Steering to Mitigate Sycophancy and Stubbornnesscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Adaptive Uncertainty-Aware Tree Search for Robust Reasoning
  2. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning
  3. 2026SuCo: Sufficiency-guided Continuous Adaptive Reasoning
  4. 2026Calibration Collapse Under Sycophancy Fine-Tuning: How Reward Hacking Breaks Uncertainty Quantification in LLMs
  5. 2025Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning
  6. 2025Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewards
  7. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
  8. 2026SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Models