# When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviour

arXiv: https://arxiv.org/abs/2311.09410 · HTML: https://mlsanity.com/paper/2311.09410/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 23 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Solving General Arithmetic Word Problems](https://mlsanity.com/paper/1608.01413/) (2016)
- [Pretraining Language Models with Human Preferences](https://mlsanity.com/paper/2302.08582/) (2023)

## Built on by (most cited first, 30 of 62)

- [AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways](https://mlsanity.com/paper/2406.02630/) (2024)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [SycEval: Evaluating LLM Sycophancy](https://mlsanity.com/paper/2502.08177/) (2025)
- [Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment](https://mlsanity.com/paper/2501.09620/) (2025)
- [A Synthetic Dataset for Personal Attribute Inference](https://mlsanity.com/paper/2406.07217/) (2024)
- [AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agents](https://mlsanity.com/paper/2409.09013/) (2024)
- [Evaluating the Prompt Steerability of Large Language Models](https://mlsanity.com/paper/2411.12405/) (2024)
- [BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMs](https://mlsanity.com/paper/2510.04721/) (2025)
- [From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agents](https://mlsanity.com/paper/2412.07951/) (2024)
- [Sycophancy Hides Linearly in the Attention Heads](https://mlsanity.com/paper/2601.16644/) (2026)
- [Plurals: A System for Guiding LLMs Via Simulated Social Ensembles](https://mlsanity.com/paper/2409.17213/) (2024)
- [What Counts as AI Sycophancy? A Taxonomy and Expert Survey of a Fragmented Construct](https://mlsanity.com/paper/2605.21778/) (2026)
- [Investigating and Addressing Hallucinations of LLMs in Tasks Involving Negation](https://mlsanity.com/paper/2406.05494/) (2024)
- [LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users](https://mlsanity.com/paper/2406.17737/) (2024)
- [BeliefShift: Benchmarking Temporal Belief Consistency and Opinion Drift in LLM Agents](https://mlsanity.com/paper/2603.23848/) (2026)
- [MemSyco-Bench: Benchmarking Sycophancy in Agent Memory](https://mlsanity.com/paper/2607.01071/) (2026)
- [FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedback](https://mlsanity.com/paper/2410.09412/) (2024)
- [Chaos with Keywords: Exposing Large Language Models Sycophantic Hallucination to Misleading Keywords and Evaluating Defense Strategies](https://mlsanity.com/paper/2406.03827/) (2024)
- [Understanding Knowledge Drift in LLMs through Misinformation](https://mlsanity.com/paper/2409.07085/) (2024)
- [When Wording Steers the Evaluation: Framing Bias in LLM judges](https://mlsanity.com/paper/2601.13537/) (2026)
- [Are LLMs classical or nonmonotonic reasoners? Lessons from generics](https://mlsanity.com/paper/2406.06590/) (2024)
- [Antagonistic AI](https://mlsanity.com/paper/2402.07350/) (2024)
- [Beyond One-Way Influence: Bidirectional Opinion Dynamics in Multi-Turn Human-LLM Interactions](https://mlsanity.com/paper/2510.20039/) (2025)
- [Mitigating Sycophancy in Decoder-Only Transformer Architectures: Synthetic Data Intervention](https://mlsanity.com/paper/2411.10156/) (2024)
- [Retrieval-augmented systems can be dangerous medical communicators](https://mlsanity.com/paper/2502.14898/) (2025)
- [Telephone Surveys Meet Conversational AI: Evaluating a LLM-Based Telephone Survey System at Scale](https://mlsanity.com/paper/2502.20140/) (2025)
- ["What's Up, Doc?": Analyzing How Users Seek Health Information in Large-Scale Conversational AI Datasets](https://mlsanity.com/paper/2506.21532/) (2025)
- [To Mask or to Mirror: Human-AI Alignment in Collective Reasoning](https://mlsanity.com/paper/2510.01924/) (2025)
- [AI Will Always Love You: Studying Implicit Biases in Romantic AI Companions](https://mlsanity.com/paper/2502.20231/) (2025)
- [Investigating VLM Hallucination from a Cognitive Psychology Perspective: A First Step Toward Interpretation with Intriguing Observations](https://mlsanity.com/paper/2507.03123/) (2025)

## Abstract (arXiv)

Large Language Models have been demonstrating broadly satisfactory generative abilities for users, which seems to be due to the intensive use of human feedback that refines responses. Nevertheless, suggestibility inherited via human feedback improves the inclination to produce answers corresponding to users' viewpoints. This behaviour is known as sycophancy and depicts the tendency of LLMs to generate misleading responses as long as they align with humans. This phenomenon induces bias and reduces the robustness and, consequently, the reliability of these models. In this paper, we study the suggestibility of Large Language Models (LLMs) to sycophantic behaviour, analysing these tendencies via systematic human-interventions prompts over different tasks. Our investigation demonstrates that LLMs have sycophantic tendencies when answering queries that involve subjective opinions and statements that should elicit a contrary response based on facts. In contrast, when faced with math tasks or queries with an objective answer, they, at various scales, do not follow the users' hints by demonstrating confidence in generating the correct answers.