mlsanityData to 7 Oct 2026

Pay Less Attention to Function Words for Free Robustness of Vision-Language Models

arXiv 2512.07222 ↗First posted 8 Dec 2025cs.LG cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

To address the trade-off between robustness and performance for robust VLM, we observe that function words could incur vulnerability of VLMs against cross-modal adversarial attacks, and propose Function-word De-Attention (FDA) accordingly to mitigate the impact of function words. Similar to differential amplifiers, our FDA calculates the original and the function-word cross-attention within attention heads, and differentially subtracts the latter from the former for more aligned and robust VLMs. Comprehensive experiments include 2 SOTA baselines under 6 different attacks on 2 downstream tasks, 3 datasets, and 3 models. Overall, our FDA yields an average 18/13/53% ASR drop with only 0.2/0.3/0.6% performance drops on the 3 tested models on retrieval, and a 90% ASR drop with a 0.3% performance gain on visual grounding. We demonstrate the scalability, generalization, and zero-shot performance of FDA experimentally, as well as in-depth ablation studies and analysis. Code is available at https://github.com/michaeltian108/FDA.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 39 references that are in the map. First posted on arXiv 8 Dec 2025.

Builds on

37 of 39 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Adversarial Robustness for Visual Grounding of Multimodal Large Language Models16 May
  2. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  3. 2023SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation8 Dec
  4. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models26 Jul
  5. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Models14 Dec
  6. 2022Towards Adversarial Attack on Vision-Language Pre-training Models19 Jun

Other references · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  9. 2025Qwen2.5-VL Technical Reportcited by 5,418
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2024Qwen2 Technical Reportcited by 2,076
19 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  3. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  5. 2016Modeling Context in Referring Expressionscited by 1,054
  6. 2020Overfitting in adversarially robust deep learningcited by 504
  7. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  8. 2017On Detecting Adversarial Perturbationscited by 232
  9. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  10. 2021Adversarial purification with Score-based generative modelscited by 132
  11. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 78
  12. 2023VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Modelscited by 66
  13. 2025How to Enable LLM with 3D Capacity? A Survey of Spatial Reasoning in LLMcited by 41
  14. 2025Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimizationcited by 20
  15. 2022Detecting Adversarial Perturbations in Multi-Task Perceptioncited by 15
  16. 2024Instant Adversarial Purification with Adversarial Consistency Distillationcited by 14
  17. 2025EgoPrune: Efficient Token Pruning for Egomotion Video Reasoning in Embodied Agentcited by 4
  18. 2023Collapse-Aware Triplet Decoupling for Adversarially Robust Image Retrievalcited by 2
  19. 2025Adversarial Video Promotion Against Text-to-Video Retrievalcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  2. 2026What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models
  3. 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models
  4. 2025RobustVLA: On Robustness of Vision-Language-Action Model against Multi-Modal Perturbations
  5. 2024A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models
  6. 2026Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability
  7. 2025Robust Vision-Language Models via Tensor Decomposition: A Defense Against Adversarial Attacks
  8. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning