# SWAY: A Counterfactual Computational Linguistic Approach to Measuring and Mitigating Sycophancy

arXiv: https://arxiv.org/abs/2604.02423 · HTML: https://mlsanity.com/paper/2604.02423/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 45 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Builds on (most cited first, 30 of 36)

- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Persona Vectors: Monitoring and Controlling Character Traits in Language Models](https://mlsanity.com/paper/2507.21509/) (2025)
- [Simple synthetic data reduces sycophancy in large language models](https://mlsanity.com/paper/2308.03958/) (2023)
- [ELEPHANT: Measuring and understanding social sycophancy in LLMs](https://mlsanity.com/paper/2505.13995/) (2025)
- [A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions](https://mlsanity.com/paper/2412.05563/) (2024)
- [SycEval: Evaluating LLM Sycophancy](https://mlsanity.com/paper/2502.08177/) (2025)
- [Measuring Sycophancy of Language Models in Multi-turn Dialogues](https://mlsanity.com/paper/2505.23840/) (2025)
- [Sycophancy in Large Language Models: Causes and Mitigations](https://mlsanity.com/paper/2411.15287/) (2024)
- [Explaining The Efficacy of Counterfactually Augmented Data](https://mlsanity.com/paper/2010.02114/) (2020)
- [Grounding Gaps in Language Model Generations](https://mlsanity.com/paper/2311.09144/) (2023)
- [AnthroScore: A Computational Linguistic Measure of Anthropomorphism](https://mlsanity.com/paper/2402.02056/) (2024)
- [Interaction Context Often Increases Sycophancy in LLMs](https://mlsanity.com/paper/2509.12517/) (2025)
- [Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI](https://mlsanity.com/paper/2511.01689/) (2025)
- [Accounting for Sycophancy in Language Model Uncertainty Estimation](https://mlsanity.com/paper/2410.14746/) (2024)
- [Sycophancy under Pressure: Evaluating and Mitigating Sycophantic Bias via Adversarial Dialogues in Scientific QA](https://mlsanity.com/paper/2508.13743/) (2025)
- [Challenging the Evaluator: LLM Sycophancy Under User Rebuttal](https://mlsanity.com/paper/2509.16533/) (2025)
- [Counterfactual Simulation Training for Chain-of-Thought Faithfulness](https://mlsanity.com/paper/2602.20710/) (2026)
- [Flattering to Deceive: The Impact of Sycophantic Behavior on User Trust in Large Language Model](https://mlsanity.com/paper/2412.02802/) (2024)
- [Calibrating Verbalized Confidence with Self-Generated Distractors](https://mlsanity.com/paper/2509.25532/) (2025)
- [Ask don't tell: Reducing sycophancy in large language models](https://mlsanity.com/paper/2602.23971/) (2026)
- [Uncertainty is Fragile: Manipulating Uncertainty in Large Language Models](https://mlsanity.com/paper/2407.11282/) (2024)
- [DebateQA: Evaluating Question Answering on Debatable Knowledge](https://mlsanity.com/paper/2408.01419/) (2024)
- [A Rational Analysis of the Effects of Sycophantic AI](https://mlsanity.com/paper/2602.14270/) (2026)
- [Accommodation and Epistemic Vigilance: A Pragmatic Account of Why LLMs Fail to Challenge Harmful Beliefs](https://mlsanity.com/paper/2601.04435/) (2026)
- [The Pluralistic Moral Gap: Understanding Judgment and Value Differences between Humans and Large Language Models](https://mlsanity.com/paper/2507.17216/) (2025)
- [Old Habits Die Hard: How Conversational History Geometrically Traps LLMs](https://mlsanity.com/paper/2603.03308/) (2026)
- [The Social Sycophancy Scale: A psychometrically validated measure of sycophancy](https://mlsanity.com/paper/2603.15448/) (2026)
- [Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning](https://mlsanity.com/paper/2512.02914/) (2025)
- [SycoEval-EM: Sycophancy Evaluation of Large Language Models in Simulated Clinical Encounters for Emergency Care](https://mlsanity.com/paper/2601.16529/) (2026)
- [Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations](https://mlsanity.com/paper/2603.03332/) (2026)

## Built on by (most cited first, 8 of 8)

- [Detecting and Controlling Sycophancy with Cascading Linear Features](https://mlsanity.com/paper/2606.26155/) (2026)
- [Receptiveness, Not Sycophancy: Distinguishing Engagement from Deference in Language Models](https://mlsanity.com/paper/2609.26579/) (2026)
- [Resist and Update: Counterfactual Report Coordinates for Incentive-Compatible LLMs](https://mlsanity.com/paper/2607.12985/) (2026)
- [This Is a Good Paper, Right? How Tag Questions Affect Sycophancy Across 106 Language Models](https://mlsanity.com/paper/2607.23976/) (2026)
- [Agents Don't Just Agree, They Remember: Benchmarking Persistent Sycophancy in Self-Improving Personal Agents](https://mlsanity.com/paper/2607.10526/) (2026)
- [Measuring and Detecting Harmful AI Sycophancy](https://mlsanity.com/paper/2608.05624/) (2026)
- [AfriSyCo: Measuring Assertive Framing, Verification, and Wording Sensitivity Around African-Language Content](https://mlsanity.com/paper/2609.17853/) (2026)
- [Pushback in a Haystack: Language Models Respond to Factual Errors Ambiguously and Ineffectively](https://mlsanity.com/paper/2610.17703/) (2026)

## Abstract (arXiv)

Large language models exhibit sycophancy: the tendency to shift outputs toward user-expressed stances, regardless of correctness or consistency. While prior work has studied this issue and its impacts, rigorous computational linguistic metrics are needed to identify when models are being sycophantic. Here, we introduce SWAY, an unsupervised computational linguistic measure of sycophancy. We develop a counterfactual prompting mechanism to identify how much a model's agreement shifts under positive versus negative linguistic pressure, isolating framing effects from content. Applying this metric to benchmark 6 models, we find that sycophancy increases with epistemic commitment. Leveraging our metric, we introduce a counterfactual mitigation strategy teaching models to consider what the answer would be if opposite assumptions were suggested. While baseline mitigation instructing to be explicitly anti-sycophantic yields moderate reductions, and can backfire, our counterfactual CoT mitigation drives sycophancy to near zero across models, commitment levels, and clause types, while not suppressing responsiveness to genuine evidence. Overall, we contribute a metric for benchmarking sycophancy and a mitigation informed by it.