mlsanityData to 14 Oct 2026

TRACE: Transparent Retrieval for Abstract Concept Evaluation

arXiv 2609.26168 ↗First posted 11 Aug 2026cs.HC cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent work reports that vision--language models (VLMs) struggle to establish and maintain stable reference in repeated reference games. Rather than ask which VLM does best, we ask a more basic question: do you need a large pretrained VLM for this at all? On grounding a single director utterance to one of twelve tangram silhouettes, we compare six off-the-shelf VLMs against a transparent baseline that uses no learned visual representation: classical SIFT keypoint matching and a signal-quality index over retrieved images. On identical trials, the transparent baseline matches the strongest VLM (SigLIP-large) and significantly outperforms the other five, including every CLIP and OpenCLIP variant. The baseline additionally retrieves external images, so this is not a matched-information comparison; what it shows is that a learned visual representation is not the bottleneck for this task: given retrieved images, a shape-appropriate classical similarity suffices. Along the way we find that abstract-grounding ability varies widely across VLMs (15--39% top-1; chance 8.33%, humans ≈77--80%), so the weakness is model-specific rather than intrinsic to contrastive pretraining; on the 1,013-shape KiloGram benchmark the pattern generalizes for CLIP, with per-shape difficulty tracking human shape-nameability. The pipeline is a classical, inspectable alternative rather than a learned one. We close by sketching how an explicit, inspectable representation of listener-side pact state could carry this approach into interactive multi-turn reference, which we leave to future work. Code available in supplementary material.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 33 references that are in the map. First posted on arXiv 11 Aug 2026.

Builds on

13 of 33 references.

References · 13

  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  2. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  3. 2020Emergent Multi-Agent Communication in the Deep Learning Eracited by 170
  4. 2022Abstract Visual Reasoning with Tangram Shapescited by 43
  5. 2019Characterizing the dynamics of learning in repeated reference gamescited by 34
  6. 2022ComCLIP: Training-Free Compositional Image and Text Matchingcited by 26
  7. 2025Measuring How (Not Just Whether) VLMs Build Common Groundcited by 8
  8. 2026LVLMs and Humans Ground Differently in Referential Communicationcited by 8
  9. 2024Language Evolution with Deep Learningcited by 5
  10. 2026Gesturing Toward Abstraction: Multimodal Convention Formation in Collaborative Physical Taskscited by 4
  11. 2025The Perceptual Observatory Characterizing Robustness and Grounding in MLLMscited by 3
  12. 2025Compositional Image-Text Matching and Retrieval by Grounding Entitiescited by 1
1 more
  1. 2026A Multimodal Framework for Aligning Human Linguistic Descriptions with Visual Perceptual Datacited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarking Deflection and Hallucination in Large Vision-Language Models
  2. 2023Revisiting the Role of Language Priors in Vision-Language Models
  3. 2024Q-GroundCAM: Quantifying Grounding in Vision Language Models via GradCAM
  4. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation
  5. 2024The Neglected Tails in Vision-Language Models
  6. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
  7. 2024Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling
  8. 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?