mlsanityData to 7 Oct 2026

Directional Embedding Smoothing for Robust Vision Language Models

arXiv 2603.15259 ↗First posted 16 Mar 2026cs.LG cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 2 more
Abstract (arXiv)

The safety and reliability of vision-language models (VLMs) are a crucial part of deploying trustworthy agentic AI systems. However, VLMs remain vulnerable to jailbreaking attacks that undermine their safety alignment to yield harmful outputs. In this work, we extend the Randomized Embedding Smoothing and Token Aggregation (RESTA) defense to VLMs and evaluate its performance against the JailBreakV-28K benchmark of multi-modal jailbreaking attacks. We find that RESTA is effective in reducing attack success rate over this diverse corpus of attacks, in particular, when employing directional embedding noise, where the injected noise is aligned with the original token embedding vectors. Our results demonstrate that RESTA can contribute to securing VLMs within agentic systems, as a lightweight, inference-time defense layer of an overall security framework.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 43 references that are in the map. First posted on arXiv 16 Mar 2026.

Builds on

33 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Understanding and Rectifying Safety Perception Distortion in VLMs18 Feb
  2. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr

Other references · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2025Gemma 3 Technical Reportcited by 1,686
  9. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  11. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  12. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
19 more
  1. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  2. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  3. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  4. 2023Detecting Language Model Attacks with Perplexitycited by 411
  5. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  6. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  7. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  8. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothingcited by 68
  9. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 65
  10. 2024Query-Based Adversarial Prompt Generationcited by 55
  11. 2024PARDEN, Can You Repeat That? Defending against Jailbreaks via Repetitioncited by 29
  12. 2023Certified Robustness for Large Language Models with Self-Denoisingcited by 23
  13. 2025STACK: Adversarial Attacks on LLM Safeguard Pipelinescited by 14
  14. 2025MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agentscited by 12
  15. 2023Randomized Smoothing with Masked Inference for Adversarially Robust Text Classificationscited by 11
  16. 2025Invitation Is All You Need! Promptware Attacks Against LLM-Powered Assistants in Production Are Practical and Dangerouscited by 11
  17. 2025Smoothed Embeddings for Robust Language Modelscited by 8
  18. 2025Chain-of-Thought Driven Adversarial Scenario Extrapolation for Robust Language Modelscited by 4
  19. 2025VERA: Variational Inference Framework for Jailbreaking Large Language Modelscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models
  2. 2026Architectural Backdoors in Vision-Language Model Supply Chains via Representation Steering
  3. 2025CeTAD: Towards Certified Toxicity-Aware Distance in Vision Language Models
  4. 2025AttackVLA: Benchmarking Adversarial and Backdoor Attacks on Vision-Language-Action Models
  5. 2025Dynamic Token Reweighting for Robust Vision-Language Models
  6. 2026VLMShield: Efficient and Robust Defense of Vision-Language Models against Malicious Prompts
  7. 2026Visual Token Compression Enhances Robustness of MLLMs
  8. 2026Less Is More -- Until It Breaks: Security Pitfalls of Vision Token Compression in Large Vision-Language Models