# TRACE: Transparent Retrieval for Abstract Concept Evaluation

arXiv: https://arxiv.org/abs/2609.26168 · HTML: https://mlsanity.com/paper/2609.26168/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 33 references that are in the map.
- First posted on arXiv 11 Aug 2026.

## Builds on (most cited first, 13 of 13)

- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Emergent Multi-Agent Communication in the Deep Learning Era](https://mlsanity.com/paper/2006.02419/) (2020)
- [Abstract Visual Reasoning with Tangram Shapes](https://mlsanity.com/paper/2211.16492/) (2022)
- [Characterizing the dynamics of learning in repeated reference games](https://mlsanity.com/paper/1912.07199/) (2019)
- [ComCLIP: Training-Free Compositional Image and Text Matching](https://mlsanity.com/paper/2211.13854/) (2022)
- [Measuring How (Not Just Whether) VLMs Build Common Ground](https://mlsanity.com/paper/2509.03805/) (2025)
- [LVLMs and Humans Ground Differently in Referential Communication](https://mlsanity.com/paper/2601.19792/) (2026)
- [Language Evolution with Deep Learning](https://mlsanity.com/paper/2403.11958/) (2024)
- [Gesturing Toward Abstraction: Multimodal Convention Formation in Collaborative Physical Tasks](https://mlsanity.com/paper/2602.08914/) (2026)
- [The Perceptual Observatory Characterizing Robustness and Grounding in MLLMs](https://mlsanity.com/paper/2512.15949/) (2025)
- [Compositional Image-Text Matching and Retrieval by Grounding Entities](https://mlsanity.com/paper/2505.02278/) (2025)
- [A Multimodal Framework for Aligning Human Linguistic Descriptions with Visual Perceptual Data](https://mlsanity.com/paper/2602.19562/) (2026)

## Abstract (arXiv)

Recent work reports that vision--language models (VLMs) struggle to establish and maintain stable reference in repeated reference games. Rather than ask which VLM does best, we ask a more basic question: do you need a large pretrained VLM for this at all? On grounding a single director utterance to one of twelve tangram silhouettes, we compare six off-the-shelf VLMs against a transparent baseline that uses no learned visual representation: classical SIFT keypoint matching and a signal-quality index over retrieved images. On identical trials, the transparent baseline matches the strongest VLM (SigLIP-large) and significantly outperforms the other five, including every CLIP and OpenCLIP variant. The baseline additionally retrieves external images, so this is not a matched-information comparison; what it shows is that a learned visual representation is not the bottleneck for this task: given retrieved images, a shape-appropriate classical similarity suffices. Along the way we find that abstract-grounding ability varies widely across VLMs (15--39% top-1; chance 8.33%, humans ≈77--80%), so the weakness is model-specific rather than intrinsic to contrastive pretraining; on the 1,013-shape KiloGram benchmark the pattern generalizes for CLIP, with per-shape difficulty tracking human shape-nameability. The pipeline is a classical, inspectable alternative rather than a learned one. We close by sketching how an explicit, inspectable representation of listener-side pact state could carry this approach into interactive multi-turn reference, which we leave to future work. Code available in supplementary material.