mlsanityData to 7 Oct 2026

Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs

arXiv 2609.09124 ↗First posted 8 Sep 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Visual encoders construct a representation of the image input for Vision-Language models. How much conceptual, as opposed to immediately visible, information does this representation contain? We use canonical color as a controlled test case to ask whether vision encoders make canonical-color information linearly accessible, even when color is removed from the input image. We construct a dataset of objects with canonical colors, and probe vision encoders for both color and object identity using color and grayscale images. We find that canonical color remains decodable from grayscale images, and is tied to predicted object identity, indicating a conceptual link. Extending this analysis to full VLMs, we find that VLM post-training can have a surprisingly large effect on color decodability in the vision encoder. Overall, canonical color provides a usefully controllable lens for tracing object-level conceptual semantic information in vision encoders and VLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 27 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

13 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts21 May
  2. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations1 Jul

Other references · 11

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  6. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  7. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  8. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  9. 2019Designing and Interpreting Probes with Control Taskscited by 608
  10. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  11. 2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustnesscited by 28

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Color Names in Vision-Language Models
  2. 2024VLM's Eye Examination: Instruct and Inspect Visual Competency of Vision Language Models
  3. 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors
  4. 2024CoLLaVO: Crayon Large Language and Vision mOdel
  5. 2024Do Pre-trained Vision-Language Models Encode Object States?
  6. 2026Grid2Matrix: Revealing Digital Agnosia in Vision-Language Models
  7. 2024Large Language Models estimate fine-grained human color-concept associations
  8. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding