mlsanityData to 14 Oct 2026

CeTAD: Towards Certified Toxicity-Aware Distance in Vision Language Models

arXiv 2503.10661 ↗First posted 8 Mar 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Recent advances in large vision-language models (VLMs) have demonstrated remarkable success across a wide range of visual understanding tasks. However, the robustness of these models against jailbreak attacks remains an open challenge. In this work, we propose a universal certified defence framework to safeguard VLMs rigorously against potential visual jailbreak attacks. First, we proposed a novel distance metric to quantify semantic discrepancies between malicious and intended responses, capturing subtle differences often overlooked by conventional cosine similarity-based measures. Then, we devise a regressed certification approach that employs randomized smoothing to provide formal robustness guarantees against both adversarial and structural perturbations, even under black-box settings. Complementing this, our feature-space defence introduces noise distributions (e.g., Gaussian, Laplacian) into the latent embeddings to safeguard against both pixel-level and structure-level perturbations. Our results highlight the potential of a formally grounded, integrated strategy toward building more resilient and trustworthy VLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 51 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

32 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation14 Mar
  2. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting14 Mar
  3. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance5 Jan
  4. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models29 Nov
  5. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  6. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul
  7. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May

Other references · 25

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  5. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  6. 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 744
  7. 2020Self-supervised Learning: Generative or Contrastivecited by 694
  8. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 567
  9. 2020A Survey on Contrastive Self-supervised Learningcited by 554
  10. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 468
  11. 2020Keystroke Dynamics as Part of Lifeloggingcited by 334
  12. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
13 more
  1. 2016Safety Verification of Deep Neural Networkscited by 263
  2. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 221
  3. 2017Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulationcited by 185
  4. 2018Certified Adversarial Robustness with Additive Noisecited by 180
  5. 2017Towards Robust Neural Networks via Random Self-ensemblecited by 159
  6. 2020Randomized Smoothing of All Shapes and Sizescited by 147
  7. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 103
  8. 2017Maximum Resilience of Artificial Neural Networkscited by 97
  9. 2019Tight Certificates of Adversarial Robustness for Randomly Smoothed Classifierscited by 65
  10. 2017Mitigating Evasion Attacks to Deep Neural Networks via Region-based Classificationcited by 56
  11. 2020Curse of Dimensionality on Randomized Smoothing for Certifiable Robustnesscited by 56
  12. 2013The cost of using exact confidence intervals for a binomial proportioncited by 11
  13. 2024Estimating the Robustness Radius for Randomized Smoothing with 100× Sample Efficiencycited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Retention Score: Quantifying Jailbreak Risks for Vision Language Models
  2. 2025Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attacks
  3. 2025Tit-for-Tat: Safeguarding Large Vision-Language Models Against Jailbreak Attacks via Adversarial Defense
  4. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  5. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreaks
  6. 2024Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models
  7. 2025JaiLIP: Jailbreaking Vision-Language Models via Loss Guided Image Perturbation
  8. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Image