mlsanityData to 14 Oct 2026

ColorBlindnessEval: Can Vision-Language Models Pass Color Blindness Tests?

arXiv 2509.19070 ↗First posted 23 Sep 2025cs.CV cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

This paper presents ColorBlindnessEval, a novel benchmark designed to evaluate the robustness of Vision-Language Models (VLMs) in visually adversarial scenarios inspired by the Ishihara color blindness test. Our dataset comprises 500 Ishihara-like images featuring numbers from 0 to 99 with varying color combinations, challenging VLMs to accurately recognize numerical information embedded in complex visual patterns. We assess 9 VLMs using Yes/No and open-ended prompts and compare their performance with human participants. Our experiments reveal limitations in the models' ability to interpret numbers in adversarial contexts, highlighting prevalent hallucination issues. These findings underscore the need to improve the robustness of VLMs in complex visual environments. ColorBlindnessEval serves as a valuable tool for benchmarking and improving the reliability of VLMs in real-world applications where accuracy is critical.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 28 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

21 of 28 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples18 Oct
  2. 2024Vision language models are blind: Failing to translate detailed visual features into words9 Jul
  3. 2023Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models3 Jun
  4. 2023Vision-Language Models for Vision Tasks: A Survey3 Apr

Other references · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  9. 2017A Survey on Deep Learning in Medical Image Analysiscited by 1,186
  10. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  11. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  12. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
5 more
  1. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Modelscited by 99
  2. 2024Benchmarking and Improving Detail Image Captioncited by 81
  3. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 75
  4. 2024ColorFoil: Investigating Color Blindness in Large Vision and Language Modelscited by 7
  5. 2023Building Safe and Reliable AI systems for Safety Critical Tasks with Vision-Language Processingcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Color Names in Vision-Language Modelscited by 2
  2. 2025What Color Is the Text? A Benchmark for Hallucination Induced by Image-Embedded Promptcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness
  2. 2025Assessing Color Vision Test in Large Vision-language Models
  3. 2025Diagnosing Vision Language Models' Perception by Leveraging Human Methods for Color Vision Deficiencies
  4. 2024VLM's Eye Examination: Instruct and Inspect Visual Competency of Vision Language Models
  5. 2026Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models
  6. 2024B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructions
  7. 2025RGB-Th-Bench: A Dense benchmark for Visual-Thermal Understanding of Vision Language Models
  8. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques