mlsanityData to 14 Oct 2026

Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of Pre-trained Multimodal Representation via Text Updates

arXiv 2505.22943 ↗First posted 28 May 2025cs.CL cs.AI cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

While pre-trained multimodal representations (e.g., CLIP) have shown impressive capabilities, they exhibit significant compositional vulnerabilities leading to counterintuitive judgments. We introduce Multimodal Adversarial Compositionality (MAC), a benchmark that leverages large language models (LLMs) to generate deceptive text samples to exploit these vulnerabilities across different modalities and evaluates them through both sample-wise attack success rate and group-wise entropy-based diversity. To improve zero-shot methods, we propose a self-training approach that leverages rejection-sampling fine-tuning with diversity-promoting filtering, which enhances both attack success rate and sample diversity. Using smaller language models like Llama-3.1-8B, our approach demonstrates superior performance in revealing compositional vulnerabilities across various multimodal representations, including images, videos, and audios.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 66 of its 85 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

66 of 85 references.

References · 66

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2013Intriguing properties of neural networkscited by 5,155
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,916
38 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  3. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  5. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  6. 2017Dense-Captioning Events in Videoscited by 793
  7. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  8. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  9. 2022Large Language Models Can Self-Improvecited by 664
  10. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  11. 2017One pixel attack for fooling deep neural networkscited by 582
  12. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  13. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 491
  14. 2020Multi-modal Transformer for Video Retrievalcited by 395
  15. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  16. 2019Clotho: An Audio Captioning Datasetcited by 358
  17. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  18. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  19. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  20. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  21. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  22. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  23. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194
  24. 2016Movie Descriptioncited by 181
  25. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 165
  26. 2022Towards Adversarial Attack on Vision-Language Pre-training Modelscited by 144
  27. 2020Perceptual Adversarial Robustness: Defense Against Unseen Threat Modelscited by 137
  28. 2020Contextualized Perturbation for Textual Adversarial Attackcited by 136
  29. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  30. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 114
  31. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 108
  32. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Modelscited by 108
  33. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  34. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103
  35. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 97
  36. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 96
  37. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  38. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026FRAME: Factored Retrieval via Attribute Readouts for Object-Centric Scene Memorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Survey
  2. 2025Universal Adversarial Attack on Aligned Multimodal LLMs
  3. 2026Investigating Adversarial Robustness of Multi-modal Large Language Models
  4. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts
  5. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective
  6. 2025Robust Defense Strategies for Multimodal Contrastive Learning: Efficient Fine-tuning Against Backdoor Attacks
  7. 2025Investigating Vulnerabilities and Defenses Against Audio-Visual Attacks: A Comprehensive Survey Emphasizing Multimodal Models
  8. 2025Survey of Adversarial Robustness in Multimodal Large Language Models