# LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples

arXiv: https://arxiv.org/abs/2310.01469 · HTML: https://mlsanity.com/paper/2310.01469/

## Key facts

- Cited by 177 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 28 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Adversarial Training for Free!](https://mlsanity.com/paper/1904.12843/) (2019)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [Sources of Hallucination by Large Language Models on Inference Tasks](https://mlsanity.com/paper/2305.14552/) (2023)
- [Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation](https://mlsanity.com/paper/2307.11019/) (2023)
- [Question Decomposition Improves the Faithfulness of Model-Generated Reasoning](https://mlsanity.com/paper/2307.11768/) (2023)
- [Enhancing Adversarial Defense by k-Winners-Take-All](https://mlsanity.com/paper/1905.10510/) (2019)
- [Trapping LLM Hallucinations Using Tagged Context Prompts](https://mlsanity.com/paper/2306.06085/) (2023)

## Built on by (most cited first, 30 of 177)

- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [CycleResearcher: Improving Automated Research via Automated Review](https://mlsanity.com/paper/2411.00816/) (2024)
- [Evaluation of Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2405.07437/) (2024)
- [Are self-explanations from Large Language Models faithful?](https://mlsanity.com/paper/2401.07927/) (2024)
- [The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models](https://mlsanity.com/paper/2401.03205/) (2024)
- [Time-MQA: Time Series Multi-Task Question Answering with Context Enhancement](https://mlsanity.com/paper/2503.01875/) (2025)
- [Alleviating Hallucinations of Large Language Models through Induced Hallucinations](https://mlsanity.com/paper/2312.15710/) (2023)
- [Determinants of LLM-assisted Decision-Making](https://mlsanity.com/paper/2402.17385/) (2024)
- [Attribute First, then Generate: Locally-attributable Grounded Text Generation](https://mlsanity.com/paper/2403.17104/) (2024)
- [LLM Voting: Human Choices and AI Collective Decision Making](https://mlsanity.com/paper/2402.01766/) (2024)
- [Multicalibration for Confidence Scoring in LLMs](https://mlsanity.com/paper/2404.04689/) (2024)
- [Confabulation: The Surprising Value of Large Language Model Hallucinations](https://mlsanity.com/paper/2406.04175/) (2024)
- [LLM-Powered User Simulator for Recommender System](https://mlsanity.com/paper/2412.16984/) (2024)
- [Exploring Autonomous Agents through the Lens of Large Language Models: A Review](https://mlsanity.com/paper/2404.04442/) (2024)
- [Grounding Language Model with Chunking-Free In-Context Retrieval](https://mlsanity.com/paper/2402.09760/) (2024)
- [Content Moderation by LLM: From Accuracy to Legitimacy](https://mlsanity.com/paper/2409.03219/) (2024)
- [DRDT: Dynamic Reflection with Divergent Thinking for LLM-based Sequential Recommendation](https://mlsanity.com/paper/2312.11336/) (2023)
- [HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wild](https://mlsanity.com/paper/2403.04307/) (2024)
- [Octopus: On-device language model for function calling of software APIs](https://mlsanity.com/paper/2404.01549/) (2024)
- [Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer](https://mlsanity.com/paper/2502.12964/) (2025)
- [FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learning](https://mlsanity.com/paper/2406.00645/) (2024)
- [Security of AI Agents](https://mlsanity.com/paper/2406.08689/) (2024)
- [Hallucination Detection in Large Language Models with Metamorphic Relations](https://mlsanity.com/paper/2502.15844/) (2025)
- [Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey](https://mlsanity.com/paper/2409.11564/) (2024)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [VeriPlan: Integrating Formal Verification and LLMs into End-User Planning](https://mlsanity.com/paper/2502.17898/) (2025)
- [Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.14891/) (2025)
- [KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality](https://mlsanity.com/paper/2506.19807/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs), including GPT-3.5, LLaMA, and PaLM, seem to be knowledgeable and able to adapt to many tasks. However, we still cannot completely trust their answers, since LLMs suffer from hallucinationtextemdash fabricating non-existent facts, deceiving users with or without their awareness. However, the reasons for their existence and pervasiveness remain unclear. In this paper, we demonstrate that nonsensical prompts composed of random tokens can also elicit the LLMs to respond with hallucinations. Moreover, we provide both theoretical and experimental evidence that transformers can be manipulated to produce specific pre-define tokens by perturbing its input sequence. This phenomenon forces us to revisit that hallucination may be another view of adversarial examples, and it shares similar characteristics with conventional adversarial examples as a basic property of LLMs. Therefore, we formalize an automatic hallucination triggering method as the textithallucination attack in an adversarial way. Finally, we explore the basic properties of attacked adversarial prompts and propose a simple yet effective defense strategy. Our code is released on GitHubfootnotehttps://github.com/PKU-YuanGroup/Hallucination-Attack.