# Dyslexify: A Mechanistic Defense Against Typographic Attacks in CLIP

arXiv: https://arxiv.org/abs/2508.20570 · HTML: https://mlsanity.com/paper/2508.20570/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- First posted on arXiv 28 Aug 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)
- LLM safety and security / [Vision-language and text-to-image safety](https://mlsanity.com/safety/vision-language-and-text-to-image-safety/)

## Built on by (most cited first, 9 of 9)

- [Read or Ignore? A Unified Benchmark for Typographic-Attack Robustness and Text Recognition in Vision-Language Models](https://mlsanity.com/paper/2512.11899/) (2025)
- [Where Vision Becomes Text: Locating the OCR Routing Bottleneck in Vision-Language Models](https://mlsanity.com/paper/2602.22918/) (2026)
- [Seeing Through Circuits: Faithful Mechanistic Interpretability for Vision Transformers](https://mlsanity.com/paper/2604.14477/) (2026)
- [A Systematic Study of Cross-Modal Typographic Attacks on Audio-Visual Reasoning](https://mlsanity.com/paper/2604.03995/) (2026)
- [Don't Look at the Numbers: Visual Anchoring Bias and Layer-wise Representation in VLMs](https://mlsanity.com/paper/2605.11218/) (2026)
- [Not What You Asked For: Typographic Attacks in Household Robot Manipulation](https://mlsanity.com/paper/2605.18593/) (2026)
- [Towards Robustness against Typographic Attack with Training-free Concept Localization](https://mlsanity.com/paper/2607.02494/) (2026)
- [QuISE: Defense against Typographic Attacks on VLMs via Query-Irrelevant Semantic Editing](https://mlsanity.com/paper/2608.13119/) (2026)
- [Detect and Suppress: A Mechanistic Defense against Adversarial Patches in VLA Models](https://mlsanity.com/paper/2610.03498/) (2026)

## Abstract (arXiv)

Typographic attacks exploit multi-modal systems by injecting text into images, leading to targeted misclassifications, malicious content generation and even Vision-Language Model jailbreaks. In this work, we analyze how CLIP vision encoders behave under typographic attacks, locating specialized attention heads in the latter half of the model's layers that causally extract and transmit typographic information to the cls token. Building on these insights, we introduce Dyslexify - a method to defend CLIP models against typographic attacks by selectively ablating a typographic circuit, consisting of attention heads. Without requiring finetuning, dyslexify improves performance by up to 22.06% on a typographic variant of ImageNet-100, while reducing standard ImageNet-100 accuracy by less than 1%, and demonstrate its utility in a medical foundation model for skin lesion diagnosis. Notably, our training-free approach remains competitive with current state-of-the-art typographic defenses that rely on finetuning. To this end, we release a family of dyslexic CLIP models which are significantly more robust against typographic attacks. These models serve as suitable drop-in replacements for a broad range of safety-critical applications, where the risks of text-based manipulation outweigh the utility of text recognition.