# BASIL: Bayesian Assessment of Sycophancy in LLMs

arXiv: https://arxiv.org/abs/2508.16846 · HTML: https://mlsanity.com/paper/2508.16846/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 48 references that are in the map.
- First posted on arXiv 23 Aug 2025.

## Builds on (most cited first, 24 of 24)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [Deal or No Deal? End-to-End Learning for Negotiation Dialogues](https://mlsanity.com/paper/1706.05125/) (2017)
- [Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good](https://mlsanity.com/paper/1906.06725/) (2019)
- [Decoupling Strategy and Generation in Negotiation Dialogues](https://mlsanity.com/paper/1808.09637/) (2018)
- [Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences](https://mlsanity.com/paper/2012.15738/) (2020)
- [CLadder: Assessing Causal Reasoning in Language Models](https://mlsanity.com/paper/2312.04350/) (2023)
- [Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment](https://mlsanity.com/paper/2311.08596/) (2023)
- [Conversations Gone Awry: Detecting Early Signs of Conversational Failure](https://mlsanity.com/paper/1805.05345/) (2018)
- [CaSiNo: A Corpus of Campsite Negotiation Dialogues for Automatic Negotiation Systems](https://mlsanity.com/paper/2103.15721/) (2021)
- [From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning](https://mlsanity.com/paper/2409.01658/) (2024)
- [NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models](https://mlsanity.com/paper/2404.12464/) (2024)
- [Steering Without Side Effects: Improving Post-Deployment Control of Language Models](https://mlsanity.com/paper/2406.15518/) (2024)
- [Trouble on the Horizon: Forecasting the Derailment of Online Conversations as they Develop](https://mlsanity.com/paper/1909.01362/) (2019)
- [Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models](https://mlsanity.com/paper/2503.17523/) (2025)
- [Human-like Few-Shot Learning via Bayesian Reasoning over Natural Language](https://mlsanity.com/paper/2306.02797/) (2023)
- [Accounting for Sycophancy in Language Model Uncertainty Estimation](https://mlsanity.com/paper/2410.14746/) (2024)
- [QUITE: Quantifying Uncertainty in Natural Language Text in Bayesian Reasoning Scenarios](https://mlsanity.com/paper/2410.10449/) (2024)
- [Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Models](https://mlsanity.com/paper/2402.03284/) (2024)

## Built on by (most cited first, 13 of 13)

- [What Counts as AI Sycophancy? A Taxonomy and Expert Survey of a Fragmented Construct](https://mlsanity.com/paper/2605.21778/) (2026)
- [Good Arguments Against the People Pleasers: How Reasoning Mitigates (Yet Masks) LLM Sycophancy](https://mlsanity.com/paper/2603.16643/) (2026)
- [BayesBench: Evaluating LLM Belief Trajectories Under Multi-Turn Evidence Accumulation](https://mlsanity.com/paper/2606.30850/) (2026)
- [Position: agentic AI orchestration should be Bayes-consistent](https://mlsanity.com/paper/2605.00742/) (2026)
- [Sycophancy Suppression Can Impair Rational Updating: Anti-Sycophancy Should Preserve the Ability to Update](https://mlsanity.com/paper/2608.26511/) (2026)
- [Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs](https://mlsanity.com/paper/2606.16011/) (2026)
- [Resist and Update: Counterfactual Report Coordinates for Incentive-Compatible LLMs](https://mlsanity.com/paper/2607.12985/) (2026)
- [Caved or Convinced: Temporal Sampling Gates Claim Deference in Video Large Language Models](https://mlsanity.com/paper/2608.03160/) (2026)
- [Playing games with knowledge: AI-Induced delusions need game theoretic interventions](https://mlsanity.com/paper/2605.08409/) (2026)
- [Pander Score: A Continuous Measure of Sycophancy as Epistemic Deference](https://mlsanity.com/paper/2606.07897/) (2026)
- [Bayesian Intelligence from the Outside](https://mlsanity.com/paper/2609.14724/) (2026)
- [Resist, Update, Reject: Preference Optimization Installs a Prior-Dependent Reliability Switch](https://mlsanity.com/paper/2609.22359/) (2026)
- [RelLedger: Relational Information Governance for LLM Agent Organizations](https://mlsanity.com/paper/2610.16281/) (2026)

## Abstract (arXiv)

Sycophancy (overly agreeable or flattering behavior) poses a fundamental challenge for human-AI collaboration, particularly in high-stakes decision-making domains such as health, law, and education. A central difficulty in studying sycophancy in large language models (LLMs) is disentangling sycophantic belief shifts from rational changes in behavior driven by new evidence or user-provided information. Existing approaches either measure descriptive behavior changes or apply normative evaluations that rely on objective ground truth, limiting their applicability to subjective or uncertain tasks. We introduce a Bayesian probabilistic framework, grounded in behavioral economics and rational decision theory, that explicitly separates sycophancy from rational belief updating. Within this framework, we achieve three objectives: (i) a descriptive metric that measures sycophancy while controlling for rational responses to evidence; (ii) a normative metric that quantifies how sycophancy leads models astray from Bayesian-consistent belief updating; and (iii) the ability to apply both metrics in settings without ground-truth labels. Applying our framework across multiple LLMs and three uncertainty-driven tasks, we find robust evidence of sycophantic belief shifts and show that their impact on rationality depends on whether models systematically over- or under-update their beliefs. Finally, we demonstrate that a post-hoc calibration method and two fine-tuning strategies (SFT and DPO) substantially reduce Bayesian inconsistency, with particularly strong improvements under explicit sycophancy prompting.