mlsanityData to 14 Oct 2026

Breaking the Illusion: Consensus-Based Generative Mitigation of Adversarial Illusions in Multi-Modal Embeddings

arXiv 2511.21893 ↗First posted 26 Nov 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Multi-modal foundation models align images, text, and other modalities in a shared embedding space but remain vulnerable to adversarial illusions [35], where imperceptible perturbations disrupt cross-modal alignment and mislead downstream tasks. To counteract the effects of adversarial illusions, we propose a task-agnostic mitigation mechanism that purifies the attacker's perturbed input using generative models, e.g., Variational Autoencoders (VAEs), to restore natural alignment. To further enhance the defense mechanism, we adopt a generative sampling strategy combined with a consensus-based aggregation scheme over the outcomes of the generated samples. Our experiments on ImageBind, a state-of-the-art multi-modal encoder, show that our approach substantially reduces the illusion attack success rates to near-zero and improves cross-modal alignment in unperturbed and perturbed input settings, providing an effective and task-agnostic defense against adversarial illusions. The code is available at https://github.com/fatemehakb/adversarial-illusions-mitigation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 26 Nov 2025.

Builds on

28 of 37 references.

References · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  7. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  8. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  9. 2017Ensemble Adversarial Training: Attacks and Defensescited by 840
  10. 2017Countering Adversarial Images using Input Transformationscited by 520
  11. 2017Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networkscited by 486
  12. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
16 more
  1. 2017MagNet: a Two-Pronged Defense against Adversarial Examplescited by 269
  2. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 124
  3. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Modelscited by 120
  4. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 120
  5. 2023AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learningcited by 103
  6. 2018Shield: Fast, Practical Defense and Vaccination for Deep Learning using JPEG Compressioncited by 74
  7. 2023Adversarial Illusions in Multi-Modal Embeddingscited by 29
  8. 2024ADBM: Adversarial diffusion bridge model for reliable adversarial purificationcited by 24
  9. 2024Adversarial Attacks to Multi-Modal Modelscited by 12
  10. 2017Adversarial Symmetric Variational Autoencodercited by 11
  11. 2025Universal Adversarial Attack on Aligned Multimodal LLMscited by 6
  12. 2025TRAP: Targeted Redirecting of Agentic Preferencescited by 5
  13. 2025Adversarial-Guided Diffusion for Multimodal LLM Attackscited by 3
  14. 2025Towards Effective and Efficient Adversarial Defense with Diffusion Models for Robust Visual Trackingcited by 2
  15. 2025Diffusion-based Cumulative Adversarial Purification for Vision Language Modelscited by 2
  16. 2025Gradient-Free Adversarial Purification with Diffusion Modelscited by 1

Built on by

Not cited yet.