mlsanityData to 13 Oct 2026

Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neurons

arXiv 2103.10480 ↗First posted 18 Mar 2021cs.LG cs.CL cs.CVcited by 29

Not on the map. Why?

Abstract (arXiv)

With Open AI's publishing of their CLIP model (Contrastive Language-Image Pre-training), multi-modal neural networks now provide accessible models that combine reading with visual recognition. Their network offers novel ways to probe its dual abilities to read text while classifying visual objects. This paper demonstrates several new categories of adversarial attacks, spanning basic typographical, conceptual, and iconographic inputs generated to fool the model into making false or absurd classifications. We demonstrate that contradictory text and image signals can confuse the model into choosing false (visual) options. Like previous authors, we show by example that the CLIP model tends to read first, look later, a phenomenon we describe as reading isn't believing.

Key facts. Cited by 29 papers in the map (data through 13 Oct 2026). Builds on 9 of its 21 references that are in the map. First posted on arXiv 18 Mar 2021.

Builds on

9 of 21 references.

References · 9

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  3. 2017Adversarial Patchcited by 516
  4. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  5. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  6. 2021Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Searchcited by 57
  7. 2019Robustness properties of Facebook's ResNeXt WSL modelscited by 17
  8. 2020Discoverability in Satellite Imagery: A Good Sentence is Worth a Thousand Picturescited by 5
  9. 2020Black Box to White Box: Discover Model Characteristics Based on Strategic Probingcited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 1; 2023: 10; 2024: 11; 2025: 6; 2026: 1.

29 papers.

Sort:

By area:

  1. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 272
  2. 2023A Pilot Study of Query-Free Adversarial Attack against Stable Diffusioncited by 76
  3. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 64
  4. 2022Contrastive language and vision learning of general fashion conceptscited by 41
  5. 2024Failures to Find Transferable Image Jailbreaks Between Vision-Language Modelscited by 28
  6. 2024Unveiling Typographic Deceptions: Insights of the Typographic Vulnerability in Large Vision-Language Modelcited by 22
  7. 2024Quantifying and Enhancing Multi-modal Robustness with Modality Preferencecited by 16
  8. 2024Whiteboard-of-Thought: Thinking Step-by-Step Across Modalitiescited by 15
  9. 2023Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificitycited by 14
  10. 2023Mutual-modality Adversarial Attack with Semantic Perturbationcited by 14
  11. 2024Espresso: Robust Concept Filtering in Text-to-Image Modelscited by 14
  12. 2023Adversarial Attacks on Foundational Vision Modelscited by 13
17 more
  1. 2024As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?cited by 11
  2. 2025Transferable Adversarial Attacks on Black-Box Vision-Language Modelscited by 10
  3. 2023Asymmetric Bias in Text-to-Image Generation with Adversarial Attackscited by 5
  4. 2024Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Modelscited by 5
  5. 2025MM-HSD: Multi-Modal Hate Speech Detection in Videoscited by 5
  6. 2024Evaluating Durability: Benchmark Insights into Multimodal Watermarkingcited by 3
  7. 2024AI Safety in Practice: Enhancing Adversarial Robustness in Multimodal Image Captioningcited by 3
  8. 2023Fooling Contrastive Language-Image Pre-trained Models with CLIPMasterPrintscited by 2
  9. 2024TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Abilitycited by 2
  10. 2025Attacking Attention of Foundation Models Disrupts Downstream Taskscited by 2
  11. 2026Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIPcited by 2
  12. 2023Black-box Attacks on Image Activity Prediction and its Natural Language Explanationscited by 1
  13. 2023Multimodal Analysis Of Google Bard And GPT-Vision: Experiments In Visual Reasoningcited by 0
  14. 2023Visual AI and Linguistic Intelligence Through Steerability and Composabilitycited by 0
  15. 2024Constructive Apraxia: An Unexpected Limit of Instructible Vision-Language Models and Analog for Human Cognitive Disorderscited by 0
  16. 2025Turning Adversaries into Allies: Reversing Typographic Attacks for Multimodal E-Commerce Product Retrievalcited by 0
  17. 2025FoCLIP: A Feature-Space Misalignment Framework for CLIP-Based Image Manipulation and Detectioncited by 0