mlsanityData to 14 Oct 2026

Tit-for-Tat: Safeguarding Large Vision-Language Models Against Jailbreak Attacks via Adversarial Defense

arXiv 2503.11619 ↗First posted 14 Mar 2025cs.CRcited by 2

On the map: LLM safety and security → Vision-language and text-to-image safety

Abstract (arXiv)

Deploying large vision-language models (LVLMs) introduces a unique vulnerability: susceptibility to malicious attacks via visual inputs. However, existing defense methods suffer from two key limitations: (1) They solely focus on textual defenses, fail to directly address threats in the visual domain where attacks originate, and (2) the additional processing steps often incur significant computational overhead or compromise model performance on benign tasks. Building on these insights, we propose ESIII (Embedding Security Instructions Into Images), a novel methodology for transforming the visual space from a source of vulnerability into an active defense mechanism. Initially, we embed security instructions into defensive images through gradient-based optimization, obtaining security instructions in the visual dimension. Subsequently, we integrate security instructions from visual and textual dimensions with the input query. The collaboration between security instructions from different dimensions ensures comprehensive security protection. Extensive experiments demonstrate that our approach effectively fortifies the robustness of LVLMs against such attacks while preserving their performance on standard benign tasks and incurring an imperceptible increase in time costs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Vision-language and text-to-image safety, one of 12 directions in LLM safety and security. Builds on 26 of its 39 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

26 of 39 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency9 Jan
  2. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt6 Jun
  3. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models14 Mar
  4. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting14 Mar
  5. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Image5 Mar
  6. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  7. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul

Other references · 19

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  8. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  9. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 183
  10. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 170
  11. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 128
  12. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 124
7 more
  1. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 117
  2. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 108
  3. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 103
  4. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 102
  5. 2023Jailbreaking GPT-4V via Self-Adversarial Attacks with System Promptscited by 65
  6. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Modelscited by 38
  7. 2023Constructing Semantics-Aware Adversarial Examples with a Probabilistic Perspectivecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025DAVSP: Safety Alignment for Large Vision-Language Models via Deep Aligned Visual Safety Promptcited by 8
  2. 2026SafeCap: Improving LVLM Safety with Image Captioning Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models
  2. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks
  3. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden States
  4. 2025TAIJI: Textual Anchoring for Immunizing Jailbreak Images in Vision Language Models
  5. 2025PRISM: Programmatic Reasoning with Image Sequence Manipulation for LVLM Jailbreaking
  6. 2025Seeing the Threat: Vulnerabilities in Vision-Language Models to Adversarial Attack
  7. 2026Jailbreaking Vision-Language Models Through the Visual Modality
  8. 2026Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking