mlsanityData to 14 Oct 2026

Chaos with Keywords: Exposing Large Language Models Sycophantic Hallucination to Misleading Keywords and Evaluating Defense Strategies

arXiv 2406.03827 ↗First posted 6 Jun 2024cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

This study explores the sycophantic tendencies of Large Language Models (LLMs), where these models tend to provide answers that match what users want to hear, even if they are not entirely correct. The motivation behind this exploration stems from the common behavior observed in individuals searching the internet for facts with partial or misleading knowledge. Similar to using web search engines, users may recall fragments of misleading keywords and submit them to an LLM, hoping for a comprehensive response. Our empirical analysis of several LLMs shows the potential danger of these models amplifying misinformation when presented with misleading keywords. Additionally, we thoroughly assess four existing hallucination mitigation strategies to reduce LLMs sycophantic behavior. Our experiments demonstrate the effectiveness of these strategies for generating factually correct statements. Furthermore, our analyses delve into knowledge-probing experiments on factual keywords and different categories of sycophancy mitigation.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 24 of its 40 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

24 of 40 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  10. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  12. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
12 more
  1. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  2. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 206
  3. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 199
  4. 2023On the Risk of Misinformation Pollution with Large Language Modelscited by 128
  5. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 108
  6. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 67
  7. 2023When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviourcited by 62
  8. 2023Augmented Large Language Models with Parametric Knowledge Guidingcited by 37
  9. 2023Large Language Models are Built-in Autoregressive Search Enginescited by 27
  10. 2021Reliability Testing for Natural Language Processing Systemscited by 25
  11. 2022HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Modelscited by 17
  12. 2023In-Context Exemplars as Clues to Retrieving from Large Associative Memorycited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 1.

10 papers.

  1. 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 104
  2. 2024Sycophancy in Large Language Models: Causes and Mitigationscited by 101
  3. 2025TRUTH DECAY: Quantifying Multi-Turn Sycophancy in Language Modelscited by 33
  4. 2024Flattering to Deceive: The Impact of Sycophantic Behavior on User Trust in Large Language Modelcited by 15
  5. 2025Sycophancy Mitigation Through Reinforcement Learning with Uncertainty-Aware Adaptive Reasoning Trajectoriescited by 11
  6. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  7. 2025Sycophancy Claims about Language Models: The Missing Human-in-the-Loopcited by 4
  8. 2026Beyond Idealized Patients: Evaluating LLMs under Challenging Patient Behaviors in Medical Consultationscited by 3
  9. 2024Are UFOs Driving Innovation? The Illusion of Causality in Large Language Modelscited by 1
  10. 2025"She was useful, but a bit too optimistic": Augmenting Design with Interactive Virtual Personascited by 0