# Seeing Is Not Addressing: Auditing Linguistic Access to Frozen Visual Geometry

arXiv: https://arxiv.org/abs/2609.37230 · HTML: https://mlsanity.com/paper/2609.37230/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 34 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality](https://mlsanity.com/paper/2306.14610/) (2023)
- [VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations](https://mlsanity.com/paper/2207.00221/) (2022)
- [FG-CLIP: Fine-Grained Visual and Textual Alignment](https://mlsanity.com/paper/2505.05071/) (2025)
- [Text-To-Concept (and Back) via Cross-Model Alignment](https://mlsanity.com/paper/2305.06386/) (2023)
- [Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset](https://mlsanity.com/paper/2004.12276/) (2020)
- [COLA: A Benchmark for Compositional Text-to-image Retrieval](https://mlsanity.com/paper/2305.03689/) (2023)
- [Text encoders bottleneck compositionality in contrastive vision-language models](https://mlsanity.com/paper/2305.14897/) (2023)
- [Interpreting the linear structure of vision-language model embedding spaces](https://mlsanity.com/paper/2504.11695/) (2025)
- [CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally](https://mlsanity.com/paper/2502.03566/) (2025)
- [Mitigate the Gap: Investigating Approaches for Improving Cross-Modal Alignment in CLIP](https://mlsanity.com/paper/2406.17639/) (2024)
- [With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You](https://mlsanity.com/paper/2506.16895/) (2025)
- [Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models](https://mlsanity.com/paper/2506.02557/) (2025)
- [LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representation](https://mlsanity.com/paper/2411.04997/) (2024)
- [Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models](https://mlsanity.com/paper/2503.17142/) (2025)
- [LOTS of Fashion! Multi-Conditioning for Image Generation via Sketch-Text Pairing](https://mlsanity.com/paper/2507.22627/) (2025)
- [Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval](https://mlsanity.com/paper/2505.15877/) (2025)
- [MERIT: Multilingual Semantic Retrieval with Interleaved Multi-Condition Query](https://mlsanity.com/paper/2506.03144/) (2025)
- [On the rankability of visual embeddings](https://mlsanity.com/paper/2507.03683/) (2025)
- [ABE-CLIP: Training-Free Attribute Binding Enhancement for Compositional Image-Text Matching](https://mlsanity.com/paper/2512.17178/) (2025)
- [Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval](https://mlsanity.com/paper/2603.01082/) (2026)
- [Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models](https://mlsanity.com/paper/2607.23052/) (2026)

## Abstract (arXiv)

Visual distinctions are often finer than those reflected in linguistic conceptualization. Vision-language models exhibit a similar asymmetry: a distinction can remain discriminable in frozen image geometry while being weakly addressable through the native text interface. We study this gap by separating visual discriminability from linguistic addressability in text-to-image retrieval. Using FactorAtlas, a fully crossed testbed of 23,040 images spanning shape, hue, pattern, and nuisance variation, we compare both readouts on held-out images of the same distinctions. We then derive image-side contrasts that separate each value from its alternatives for matched visual grounding, and test whether this reduces the native-text access gap across factors and models. Direction-specific and visual-absence controls tie these gains to the relevant visual contrast; the gains persist after global alignment and extend to compositional retrieval and natural images. Together, these results show that visual discriminability and linguistic addressability need not coincide, and that matched visual grounding can probe and reduce the resulting access gap.