# Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neurons

arXiv: https://arxiv.org/abs/2103.10480 · HTML: https://mlsanity.com/paper/2103.10480/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 18 Mar 2021.

## Builds on (most cited first, 9 of 9)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Adversarial Patch](https://mlsanity.com/paper/1712.09665/) (2017)
- [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning](https://mlsanity.com/paper/2103.01913/) (2021)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search](https://mlsanity.com/paper/2102.01645/) (2021)
- [Robustness properties of Facebook's ResNeXt WSL models](https://mlsanity.com/paper/1907.07640/) (2019)
- [Discoverability in Satellite Imagery: A Good Sentence is Worth a Thousand Pictures](https://mlsanity.com/paper/2001.05839/) (2020)
- [Black Box to White Box: Discover Model Characteristics Based on Strategic Probing](https://mlsanity.com/paper/2009.03136/) (2020)

## Built on by (most cited first, 29 of 29)

- [Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models](https://mlsanity.com/paper/2307.14539/) (2023)
- [A Pilot Study of Query-Free Adversarial Attack against Stable Diffusion](https://mlsanity.com/paper/2303.16378/) (2023)
- [Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety](https://mlsanity.com/paper/2502.05206/) (2025)
- [Contrastive language and vision learning of general fashion concepts](https://mlsanity.com/paper/2204.03972/) (2022)
- [Failures to Find Transferable Image Jailbreaks Between Vision-Language Models](https://mlsanity.com/paper/2407.15211/) (2024)
- [Unveiling Typographic Deceptions: Insights of the Typographic Vulnerability in Large Vision-Language Model](https://mlsanity.com/paper/2402.19150/) (2024)
- [Quantifying and Enhancing Multi-modal Robustness with Modality Preference](https://mlsanity.com/paper/2402.06244/) (2024)
- [Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities](https://mlsanity.com/paper/2406.14562/) (2024)
- [Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity](https://mlsanity.com/paper/2306.16048/) (2023)
- [Mutual-modality Adversarial Attack with Semantic Perturbation](https://mlsanity.com/paper/2312.12768/) (2023)
- [Espresso: Robust Concept Filtering in Text-to-Image Models](https://mlsanity.com/paper/2404.19227/) (2024)
- [Adversarial Attacks on Foundational Vision Models](https://mlsanity.com/paper/2308.14597/) (2023)
- [As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?](https://mlsanity.com/paper/2403.12693/) (2024)
- [Transferable Adversarial Attacks on Black-Box Vision-Language Models](https://mlsanity.com/paper/2505.01050/) (2025)
- [Asymmetric Bias in Text-to-Image Generation with Adversarial Attacks](https://mlsanity.com/paper/2312.14440/) (2023)
- [Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Models](https://mlsanity.com/paper/2405.20090/) (2024)
- [MM-HSD: Multi-Modal Hate Speech Detection in Videos](https://mlsanity.com/paper/2508.20546/) (2025)
- [Evaluating Durability: Benchmark Insights into Multimodal Watermarking](https://mlsanity.com/paper/2406.03728/) (2024)
- [AI Safety in Practice: Enhancing Adversarial Robustness in Multimodal Image Captioning](https://mlsanity.com/paper/2407.21174/) (2024)
- [Fooling Contrastive Language-Image Pre-trained Models with CLIPMasterPrints](https://mlsanity.com/paper/2307.03798/) (2023)
- [TIMA: Text-Image Mutual Awareness for Balancing Zero-Shot Adversarial Robustness and Generalization Ability](https://mlsanity.com/paper/2405.17678/) (2024)
- [Attacking Attention of Foundation Models Disrupts Downstream Tasks](https://mlsanity.com/paper/2506.05394/) (2025)
- [Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP](https://mlsanity.com/paper/2601.19210/) (2026)
- [Black-box Attacks on Image Activity Prediction and its Natural Language Explanations](https://mlsanity.com/paper/2310.00503/) (2023)
- [Multimodal Analysis Of Google Bard And GPT-Vision: Experiments In Visual Reasoning](https://mlsanity.com/paper/2309.16705/) (2023)
- [Visual AI and Linguistic Intelligence Through Steerability and Composability](https://mlsanity.com/paper/2312.12383/) (2023)
- [Constructive Apraxia: An Unexpected Limit of Instructible Vision-Language Models and Analog for Human Cognitive Disorders](https://mlsanity.com/paper/2410.03551/) (2024)
- [Turning Adversaries into Allies: Reversing Typographic Attacks for Multimodal E-Commerce Product Retrieval](https://mlsanity.com/paper/2511.05325/) (2025)
- [FoCLIP: A Feature-Space Misalignment Framework for CLIP-Based Image Manipulation and Detection](https://mlsanity.com/paper/2511.06947/) (2025)

## Abstract (arXiv)

With Open AI's publishing of their CLIP model (Contrastive Language-Image Pre-training), multi-modal neural networks now provide accessible models that combine reading with visual recognition. Their network offers novel ways to probe its dual abilities to read text while classifying visual objects. This paper demonstrates several new categories of adversarial attacks, spanning basic typographical, conceptual, and iconographic inputs generated to fool the model into making false or absurd classifications. We demonstrate that contradictory text and image signals can confuse the model into choosing false (visual) options. Like previous authors, we show by example that the CLIP model tends to read first, look later, a phenomenon we describe as reading isn't believing.