mlsanityData to 14 Oct 2026

Towards Understanding Sycophancy in Language Models

arXiv 2310.13548 ↗First posted 20 Oct 2023cs.CL cs.AI cs.LGcited by 1,040

Not on the map. Why?

Cited by 160 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Human feedback is commonly utilized to finetune AI assistants. But human feedback may also encourage model responses that match user beliefs over truthful ones, a behaviour known as sycophancy. We investigate the prevalence of sycophancy in models whose finetuning procedure made use of human feedback, and the potential role of human preference judgments in such behavior. We first demonstrate that five state-of-the-art AI assistants consistently exhibit sycophancy across four varied free-form text-generation tasks. To understand if human preferences drive this broadly observed behavior, we analyze existing human preference data. We find that when a response matches a user's views, it is more likely to be preferred. Moreover, both humans and preference models (PMs) prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time. Optimizing model outputs against PMs also sometimes sacrifices truthfulness in favor of sycophancy. Overall, our results indicate that sycophancy is a general behavior of state-of-the-art AI assistants, likely driven in part by human preference judgments favoring sycophantic responses.

Key facts. Cited by 1,040 papers in the map (data through 14 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

31 of 44 references.

References · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  11. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
19 more
  1. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  2. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  4. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  5. 2018AI safety via debatecited by 400
  6. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  7. 2022Self-critiquing models for assisting human evaluatorscited by 380
  8. 2023The False Promise of Imitating Proprietary LLMscited by 263
  9. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  10. 2019Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyrocited by 191
  11. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  12. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
  13. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 108
  14. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49
  15. 2017Occam's razor is insufficient to infer the preferences of irrational agentscited by 45
  16. 2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25
  17. 2022Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learningcited by 22
  18. 2020Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learningcited by 21
  19. 2016Learning Mixtures of Plackett-Luce Modelscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 124; 2025: 230; 2026: 673.

1,040 papers.

Sort:

By area:

  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  2. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  3. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  4. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  5. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  6. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 176
  7. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  8. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  9. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 160
  10. 2024Distilling System 2 into System 1cited by 159
  11. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 150
  12. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
38 more
  1. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  2. 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 129
  3. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  4. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  5. 2024Language Models Learn to Mislead Humans via RLHFcited by 124
  6. 2023Generative Language Models Exhibit Social Identity Biasescited by 111
  7. 2025SycEval: Evaluating LLM Sycophancycited by 108
  8. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  9. 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 104
  10. 2024Sycophancy in Large Language Models: Causes and Mitigationscited by 101
  11. 2025LLM Social Simulations Are a Promising Research Methodcited by 99
  12. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 96
  13. 2023System 2 Attention (is something you might need too)cited by 89
  14. 2023Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experimentcited by 77
  15. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 77
  16. 2023Alignment for Honestycited by 74
  17. 2024Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voterscited by 71
  18. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
  19. 2024Factuality of Large Language Models: A Surveycited by 66
  20. 2024A Computational Framework for Behavioral Assessment of LLM Therapistscited by 62
  21. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 62
  22. 2024DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Lifecited by 59
  23. 2026Emotion Concepts and their Function in a Large Language Modelcited by 57
  24. 2024From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuningcited by 54
  25. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimizationcited by 53
  26. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  27. 2024How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensionscited by 52
  28. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structurecited by 52
  29. 2024LatentQA: Teaching LLMs to Decode Activations Into Natural Languagecited by 51
  30. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 50
  31. 2025When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Modelscited by 50
  32. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  33. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  34. 2024A Safe Harbor for AI Evaluation and Red Teamingcited by 47
  35. 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedbackcited by 47
  36. 2025Why human-AI relationships need socioaffective alignmentcited by 47
  37. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 45
  38. 2023Debate Helps Supervise Unreliable Expertscited by 43

50 most cited of 1,040 All 1,040 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment Without Understanding: A Message- and Conversation-Centered Approach to Understanding AI Sycophancy
  2. 2026AI Revealed Preferences
  3. 2026Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment
  4. 2024Linear Probe Penalties Reduce LLM Sycophancy
  5. 2023Human Feedback is not Gold Standard
  6. 2026Token-Level Diagnosis of Sycophancy in LLMs with Attribution-Guided Steering
  7. 2026It's Not Always Sycophancy: Measuring LLM Conformity as a Function of Epistemic Uncertainty
  8. 2026AI Should Not Only Be Helpful. It Should Be Contingent. Artificial Intimacy, Sycophancy, and the Future of Social Learning