mlsanityData to 14 Oct 2026

Robust Vision-Language Models via Tensor Decomposition: A Defense Against Adversarial Attacks

arXiv 2509.16163 ↗First posted 19 Sep 2025cs.CV cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Vision language models (VLMs) excel in multimodal understanding but are prone to adversarial attacks. Existing defenses often demand costly retraining or significant architecture changes. We introduce a lightweight defense using tensor decomposition suitable for any pre-trained VLM, requiring no retraining. By decomposing and reconstructing vision encoder representations, it filters adversarial noise while preserving meaning. Experiments with CLIP on COCO and Flickr30K show improved robustness. On Flickr30K, it restores 12.3% performance lost to attacks, raising Recall@1 accuracy from 7.5% to 19.8%. On COCO, it recovers 8.1% performance, improving accuracy from 3.8% to 11.9%. Analysis shows Tensor Train decomposition with low rank (8-32) and low residual strength (α=0.1-0.2) is optimal. This method is a practical, plug-and-play solution with minimal overhead for existing VLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 22 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

13 of 22 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness16 Jan
  2. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  3. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec

Other references · 10

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  3. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 678
  4. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  5. 2017Adversarial Examples for Semantic Segmentation and Object Detectioncited by 275
  6. 2016TensorLy: Tensor Learning in Pythoncited by 104
  7. 2024Online Zero-Shot Classification with CLIPcited by 22
  8. 2021Detecting AutoAttack Perturbations in the Frequency Domaincited by 10
  9. 2024Enhancing Image Retrieval : A Comprehensive Study on Photo Search using the CLIP Modecited by 6
  10. 2023Video Question Answering Using CLIP-Guided Visual-Text Attentioncited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  2. 2026Corrupted but Correct: Why Vision-Language Models Lie to Themselves Internally
  3. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models
  4. 2026PDA: Text-Augmented Defense Framework for Robust Vision-Language Models against Adversarial Image Attacks
  5. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models
  6. 2025Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models
  7. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning
  8. 2025Diffusion-based Cumulative Adversarial Purification for Vision Language Models