mlsanityData to 7 Oct 2026

Triggering Hallucinations in LLMs: A Quantitative Study of Prompt-Induced Hallucination in Large Language Models

arXiv 2505.00557 ↗First posted 1 May 2025cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Hallucinations in large language models (LLMs) present a growing challenge across real-world applications, from healthcare to law, where factual reliability is essential. Despite advances in alignment and instruction tuning, LLMs can still generate outputs that are fluent yet fundamentally untrue. Understanding the cognitive dynamics that underlie these hallucinations remains an open problem. In this study, we propose a prompt-based framework to systematically trigger and quantify hallucination: a Hallucination-Inducing Prompt (HIP), which synthetically fuses semantically distant concepts (e.g., periodic table of elements and tarot divination) in a misleading way, and a Hallucination Quantifying Prompt (HQP), which scores the plausibility, confidence, and coherence of the output. Controlled experiments across multiple LLMs revealed that HIPs consistently produced less coherent and more hallucinated responses than their null-fusion controls. These effects varied across models, with reasoning-oriented LLMs showing distinct profiles from general-purpose ones. Our framework provides a reproducible testbed for studying hallucination vulnerability, and opens the door to developing safer, more introspective LLMs that can detect and self-regulate the onset of conceptual instability.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 9 of its 12 references that are in the map. First posted on arXiv 1 May 2025.

Builds on

9 of 12 references.

References · 9

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  4. 2022Using cognitive psychology to understand GPT-3cited by 389
  5. 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 130
  6. 2024The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Modelscited by 87
  7. 2024Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Modelscited by 84
  8. 2023Exploiting Language Models as a Source of Knowledge for Cognitive Agentscited by 15
  9. 2025Waking Up an AI: A Quantitative Framework for Prompt-Induced Phase Transition in Large Language Modelscited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026Mitigating Prompt-Induced Hallucinations in Large Language Models via Structured Reasoning
  2. 2025Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluation
  3. 2026PRISM: Probing Reasoning, Instruction, and Source Memory in LLM Hallucinations
  4. 2025From "Hallucination" to "Suture": Insights from Language Philosophy to Enhance Large Language Models
  5. 2024Investigating the Role of Prompting and External Tools in Hallucination Rates of Large Language Models
  6. 2026Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity
  7. 2025Auditing Meta-Cognitive Hallucinations in Reasoning Large Language Models
  8. 2026Evaluating Hallucinations in Domain-Adapted Large Language Models