mlsanityData to 14 Oct 2026

AMIA: Automatic Masking and Joint Intention Analysis Makes LVLMs Robust Jailbreak Defenders

arXiv 2505.24519 ↗First posted 30 May 2025cs.CV cs.CLcited by 5

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

We introduce AMIA, a lightweight, inference-only defense for Large Vision-Language Models (LVLMs) that (1) Automatically Masks a small set of text-irrelevant image patches to disrupt adversarial perturbations, and (2) conducts joint Intention Analysis to uncover and mitigate hidden harmful intents before response generation. Without any retraining, AMIA improves defense success rates across diverse LVLMs and jailbreak benchmarks from an average of 52.4% to 81.7%, preserves general utility with only a 2% average accuracy drop, and incurs only modest inference overhead. Ablation confirms both masking and intention analysis are essential for a robust safety-utility trade-off.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 33 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

20 of 33 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations14 Feb
  2. 2025Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency9 Jan
  3. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks23 Nov
  4. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time9 Oct
  5. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar
  6. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2016A Diagram Is Worth A Dozen Imagescited by 804
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  6. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  7. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  8. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  9. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 103
  10. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 79
  11. 2024Intention Analysis Makes LLMs A Good Jailbreak Defendercited by 74
  12. 2023JailGuard: A Universal Detection Framework for LLM Prompt-based Attackscited by 50
2 more
  1. 2025Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAGcited by 23
  2. 2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hackingcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025OpenLVLM-MIA: A Controlled Benchmark Revealing the Limits of Membership Inference Attacks on Large Vision-Language Modelscited by 1
  2. 2026GuardAlign: Test-time Safety Alignment in Multimodal Large Language Modelscited by 1
  3. 2026Mosaic: Multimodal Jailbreak against Closed-Source VLMs via Multi-View Ensemble Optimizationcited by 0
  4. 2026Structure-Guided Visual Perturbation Neutralization for LVLMscited by 0
  5. 2026ReFrame: Evidence-Guided Test-Time Safety Alignment in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden States
  2. 2025Tit-for-Tat: Safeguarding Large Vision-Language Models Against Jailbreak Attacks via Adversarial Defense
  3. 2025TAIJI: Textual Anchoring for Immunizing Jailbreak Images in Vision Language Models
  4. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt
  5. 2025PRISM: Programmatic Reasoning with Image Sequence Manipulation for LVLM Jailbreaking
  6. 2025Towards Effective MLLM Jailbreaking Through Balanced On-Topicness and OOD-Intensity
  7. 2025Learning to Detect Unseen Jailbreak Attacks in Large Vision-Language Models
  8. 2026Toward Universal and Transferable Jailbreak Attacks on Vision-Language Models