# Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Models

arXiv: https://arxiv.org/abs/2504.01589 · HTML: https://mlsanity.com/paper/2504.01589/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Counterfactual VQA: A Cause-Effect Look at Language Bias](https://mlsanity.com/paper/2006.04315/) (2020)
- [ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs](https://mlsanity.com/paper/2402.11753/) (2024)
- [mDPO: Conditional Preference Optimization for Multimodal Large Language Models](https://mlsanity.com/paper/2406.11839/) (2024)
- [Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement](https://mlsanity.com/paper/2405.15973/) (2024)
- [Level Generation Through Large Language Models](https://mlsanity.com/paper/2302.05817/) (2023)
- [Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models](https://mlsanity.com/paper/2404.03622/) (2024)
- [Predicting Question-Answering Performance of Large Language Models through Semantic Consistency](https://mlsanity.com/paper/2311.01152/) (2023)
- [Bot or Human? Detecting ChatGPT Imposters with A Single Question](https://mlsanity.com/paper/2305.06424/) (2023)
- [Defense-Prefix for Preventing Typographic Attacks on CLIP](https://mlsanity.com/paper/2304.04512/) (2023)
- [SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environments](https://mlsanity.com/paper/2412.00114/) (2024)
- [Self-Consistency of Large Language Models under Ambiguity](https://mlsanity.com/paper/2310.13439/) (2023)
- [Testing the Depth of ChatGPT's Comprehension via Cross-Modal Tasks Based on ASCII-Art: GPT3.5's Abilities in Regard to Recognizing and Generating ASCII-Art Are Not Totally Lacking](https://mlsanity.com/paper/2307.16806/) (2023)
- [Large language models and (non-)linguistic recursion](https://mlsanity.com/paper/2306.07195/) (2023)
- [ASCIIEval: Benchmarking Models' Visual Perception in Text Strings via ASCII Art](https://mlsanity.com/paper/2410.01733/) (2024)
- [Evading Toxicity Detection with ASCII-art: A Benchmark of Spatial Attacks on Moderation Systems](https://mlsanity.com/paper/2409.18708/) (2024)

## Built on by (most cited first, 5 of 5)

- [Latent Visual Reasoning](https://mlsanity.com/paper/2509.24251/) (2025)
- [VistaWise: Building Cost-Effective Agent with Cross-Modal Knowledge Graph for Minecraft](https://mlsanity.com/paper/2508.18722/) (2025)
- [Interpretable Debiasing of Vision-Language Models for Social Fairness](https://mlsanity.com/paper/2602.24014/) (2026)
- [Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Models](https://mlsanity.com/paper/2508.01678/) (2025)
- [From Building Blocks to Planning: Multi-Step Spatial Reasoning in LLMs with Reinforcement Learning](https://mlsanity.com/paper/2512.24532/) (2025)

## Abstract (arXiv)

Vision-language models (VLMs) have advanced rapidly in processing multimodal information, but their ability to reconcile conflicting signals across modalities remains underexplored. This work investigates how VLMs process ASCII art, a unique medium where textual elements collectively form visual patterns, potentially creating semantic-visual conflicts. We introduce a novel evaluation framework that systematically challenges five state-of-the-art models (including GPT-4o, Claude, and Gemini) using adversarial ASCII art, where character-level semantics deliberately contradict global visual patterns. Our experiments reveal a strong text-priority bias: VLMs consistently prioritize textual information over visual patterns, with visual recognition ability declining dramatically as semantic complexity increases. Various mitigation attempts through visual parameter tuning and prompt engineering yielded only modest improvements, suggesting that this limitation requires architectural-level solutions. These findings uncover fundamental flaws in how current VLMs integrate multimodal information, providing important guidance for future model development while highlighting significant implications for content moderation systems vulnerable to adversarial examples.