mlsanityData to 7 Oct 2026

LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples

arXiv 2310.01469 ↗First posted 2 Oct 2023cs.CL cs.AIcited by 177

Not on the map. Why?

Cited by 24 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs), including GPT-3.5, LLaMA, and PaLM, seem to be knowledgeable and able to adapt to many tasks. However, we still cannot completely trust their answers, since LLMs suffer from hallucinationtextemdash fabricating non-existent facts, deceiving users with or without their awareness. However, the reasons for their existence and pervasiveness remain unclear. In this paper, we demonstrate that nonsensical prompts composed of random tokens can also elicit the LLMs to respond with hallucinations. Moreover, we provide both theoretical and experimental evidence that transformers can be manipulated to produce specific pre-define tokens by perturbing its input sequence. This phenomenon forces us to revisit that hallucination may be another view of adversarial examples, and it shares similar characteristics with conventional adversarial examples as a basic property of LLMs. Therefore, we formalize an automatic hallucination triggering method as the textithallucination attack in an adversarial way. Finally, we explore the basic properties of attacked adversarial prompts and propose a simple yet effective defense strategy. Our code is released on GitHubfootnotehttps://github.com/PKU-YuanGroup/Hallucination-Attack.

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). Builds on 23 of its 28 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

23 of 28 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  12. 2023PaLM 2 Technical Reportcited by 1,154
11 more
  1. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  2. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  3. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  4. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  5. 2019Adversarial Training for Free!cited by 638
  6. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  7. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 168
  8. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 146
  9. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 107
  10. 2019Enhancing Adversarial Defense by k-Winners-Take-Allcited by 51
  11. 2023Trapping LLM Hallucinations Using Tagged Context Promptscited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 68; 2025: 80; 2026: 21.

177 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  4. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  5. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  6. 2024Are self-explanations from Large Language Models faithful?cited by 101
  7. 2024The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Modelscited by 87
  8. 2023Alleviating Hallucinations of Large Language Models through Induced Hallucinationscited by 78
  9. 2025Time-MQA: Time Series Multi-Task Question Answering with Context Enhancementcited by 74
  10. 2024Determinants of LLM-assisted Decision-Makingcited by 55
  11. 2024Attribute First, then Generate: Locally-attributable Grounded Text Generationcited by 54
  12. 2024LLM Voting: Human Choices and AI Collective Decision Makingcited by 47
38 more
  1. 2024Multicalibration for Confidence Scoring in LLMscited by 44
  2. 2024Confabulation: The Surprising Value of Large Language Model Hallucinationscited by 42
  3. 2024LLM-Powered User Simulator for Recommender Systemcited by 38
  4. 2024Grounding Language Model with Chunking-Free In-Context Retrievalcited by 30
  5. 2024Exploring Autonomous Agents through the Lens of Large Language Models: A Reviewcited by 30
  6. 2024Content Moderation by LLM: From Accuracy to Legitimacycited by 30
  7. 2023DRDT: Dynamic Reflection with Divergent Thinking for LLM-based Sequential Recommendationcited by 27
  8. 2024HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wildcited by 27
  9. 2024Octopus: On-device language model for function calling of software APIscited by 26
  10. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  11. 2025Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answercited by 26
  12. 2024Security of AI Agentscited by 24
  13. 2025Hallucination Detection in Large Language Models with Metamorphic Relationscited by 24
  14. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  15. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  16. 2026Large Language Model Reasoning Failurescited by 20
  17. 2024CloChat: Understanding How People Customize, Interact, and Experience Personas in Large Language Modelscited by 19
  18. 2024Constructing Benchmarks and Interventions for Combating Hallucinations in LLMscited by 19
  19. 2025VeriPlan: Integrating Formal Verification and LLMs into End-User Planningcited by 19
  20. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  21. 2025Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Modelscited by 18
  22. 2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 18
  23. 2024Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Artcited by 17
  24. 2024RemoteRAG: A Privacy-Preserving LLM Cloud RAG Servicecited by 17
  25. 2025Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Modelscited by 17
  26. 2025Uncertainty Quantification of Large Language Models through Multi-Dimensional Responsescited by 17
  27. 2025GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluationcited by 17
  28. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 17
  29. 2025Butterfly Effects in Toolchains: A Comprehensive Analysis of Failed Parameter Filling in LLM Tool-Agent Systemscited by 17
  30. 2024Preemptive Answer "Attacks" on Chain-of-Thought Reasoningcited by 16
  31. 2024Manipulation Facing Threats: Evaluating Physical Vulnerabilities in End-to-End Vision Language Action Modelscited by 16
  32. 2024Truth-Aware Context Selection: Mitigating Hallucinations of Large Language Models Being Misled by Untruthful Contextscited by 15
  33. 2024PiCO: Peer Review in LLMs based on the Consistency Optimizationcited by 14
  34. 2024Distinguishing Ignorance from Error in LLM Hallucinationscited by 14
  35. 2025EraRAG: Efficient and Incremental Retrieval Augmented Generation for Growing Corporacited by 14
  36. 2024DebUnc: Improving Large Language Model Agent Communication With Uncertainty Metricscited by 13
  37. 2025Thinking beyond the anthropomorphic paradigm benefits LLM researchcited by 13
  38. 2025Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Modelscited by 13

50 most cited of 177 All 177 (JSON)