The Illusionist's Prompt: Exposing the Factual Vulnerabilities of Large Language Models with Linguistic Nuances
Not on the map. Why?
Abstract (arXiv)
As Large Language Models (LLMs) continue to advance, they are increasingly relied upon as real-time sources of information by non-expert users. To ensure the factuality of the information they provide, much research has focused on mitigating hallucinations in LLM responses, but only in the context of formal user queries, rather than maliciously crafted ones. In this study, we introduce The Illusionist's Prompt, a novel hallucination attack that incorporates linguistic nuances into adversarial queries, challenging the factual accuracy of LLMs against five types of fact-enhancing strategies. Our attack automatically generates highly transferrable illusory prompts to induce internal factual errors, all while preserving user intent and semantics. Extensive experiments confirm the effectiveness of our attack in compromising black-box LLMs, including commercial APIs like GPT-4o and Gemini-2.0, even with various defensive mechanisms.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 35 of its 54 references that are in the map. First posted on arXiv 1 Apr 2025.
Builds on
35 of 54 references.
References · 35
- 2023GPT-4 Technical Reportcited by 18,132
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
- 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
- 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
- 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
- 2023How Language Model Hallucinations Can Snowballcited by 310
- 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
- 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 246
23 more
- 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
- 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
- 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
- 2025Qwen2.5-1M Technical Reportcited by 158
- 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 114
- 2023The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversationcited by 107
- 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 95
- 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 90
- 2023AlpaCare:Instruction-tuned Large Language Models for Medical Applicationcited by 88
- 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 80
- 2024Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluationcited by 79
- 2023Alleviating Hallucinations of Large Language Models through Induced Hallucinationscited by 78
- 2024Factuality of Large Language Models: A Surveycited by 64
- 2023Truth Forest: Toward Multi-Scale Truthfulness in Large Language Models through Intervention without Tuningcited by 45
- 2024Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigationcited by 43
- 2024Unveiling the Safety of GPT-4o: An Empirical Study using Jailbreak Attackscited by 35
- 2024Mitigating Large Language Model Hallucination with Faithful Finetuningcited by 24
- 2024Improving Factuality in Large Language Models via Decoding-Time Hallucinatory and Truthful Comparatorscited by 19
- 2024Comparing Hallucination Detection Metrics for Multilingual Generationcited by 18
- 2023Exploring the Relationship between LLM Hallucinations and Prompt Linguistic Nuances: Readability, Formality, and Concretenesscited by 13
- 2024FinDABench: Benchmarking Financial Data Analysis Ability of Large Language Modelscited by 11
- 2024Fine-Grained Self-Endorsement Improves Factuality and Reasoningcited by 7
- 2025Mitigating Hallucinated Translations in Large Language Models with Hallucination-focused Preference Optimizationcited by 6
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.