From Hallucinations to Jailbreaks: Rethinking the Vulnerability of Large Foundation Models
On the map: LLM safety and security → Jailbreaking
Abstract (arXiv)
Large foundation models (LFMs) are susceptible to two distinct vulnerabilities: hallucinations and jailbreak attacks. While typically studied in isolation, we observe that defenses targeting one often affect the other, hinting at a deeper connection. We propose a unified theoretical framework that models jailbreaks as token-level optimization and hallucinations as attention-level optimization. Within this framework, we establish two key propositions: (1) textitSimilar Loss Convergence - the loss functions for both vulnerabilities converge similarly when optimizing for target-specific outputs; and (2) textitGradient Consistency in Attention Redistribution - both exhibit consistent gradient behavior driven by shared attention dynamics. We validate these propositions empirically on LLaVA-1.5 and MiniGPT-4, showing consistent optimization trends and aligned gradients. Leveraging this connection, we demonstrate that mitigation techniques for hallucinations can reduce jailbreak success rates, and vice versa. Our findings reveal a shared failure mode in LFMs and suggest that robustness strategies should jointly address both vulnerabilities.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 36 of its 43 references that are in the map. First posted on arXiv 30 May 2025.
Builds on
36 of 43 references.
Closest ancestors · 10
Same direction, newest first
- 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models26 Jun
- 2024Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters30 May
- 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
- 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers25 Feb
- 2024Query-Based Adversarial Prompt Generation19 Feb
- 2024GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Models5 Feb
- 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
- 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
- 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
Other references · 26
- 2023Visual Instruction Tuningcited by 7,869
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
- 2019Are Sixteen Heads Really Better than One?cited by 816
- 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
- 2023The Internal State of an LLM Knows When It's Lyingcited by 659
- 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
- 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
- 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
- 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
- 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 323
- 2023Are aligned neural networks adversarially aligned?cited by 315
14 more
- 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
- 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
- 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 145
- 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 142
- 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 125
- 2024Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Modelscited by 86
- 2024Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categoriescited by 77
- 2023Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Modelscited by 67
- 2022SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehiclescited by 66
- 2024Jailbreak Large Vision-Language Models Through Multi-Modal Linkagecited by 62
- 2024Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden Statescited by 52
- 2024Detecting Hallucinations in Large Language Model Generation: A Token Probability Approachcited by 42
- 2024AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Modelscited by 32
- 2024InternalInspector I2: Robust Confidence Estimation in LLMs through Internal Statescited by 30
Built on by
Citing papers per year: 2025: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024"Not Aligned" is Not "Malicious": Being Careful about Hallucinations of Large Language Models' Jailbreak
- 2026Systematic Scaling Analysis of Jailbreak Attacks in Large Language Models
- 2024JailbreakLens: Interpreting Jailbreak Mechanism in the Lens of Representation and Circuit
- 2025Attention Slipping: A Mechanistic Understanding of Jailbreak Attacks and Defenses in LLMs
- 2024Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models
- 2025Breaking the Ceiling: Exploring the Potential of Jailbreak Attacks through Expanding Strategy Space
- 2026Jailbroken Frontier Models Retain Their Capabilities
- 2025Behind the Mask: Benchmarking Camouflaged Jailbreaks in Large Language Models