# Binding Visual Features Point by Point

arXiv: https://arxiv.org/abs/2605.25427 · HTML: https://mlsanity.com/paper/2605.25427/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 26 of its 48 references that are in the map.
- First posted on arXiv 25 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 26 of 26)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [Direct and Indirect Effects](https://mlsanity.com/paper/1301.2300/) (2013)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling](https://mlsanity.com/paper/2111.12085/) (2021)
- [Grounded Reinforcement Learning for Visual Reasoning](https://mlsanity.com/paper/2505.23678/) (2025)
- [Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem](https://mlsanity.com/paper/2411.00238/) (2024)
- [Testing Relational Understanding in Text-Guided Image Generation](https://mlsanity.com/paper/2208.00005/) (2022)
- [Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks](https://mlsanity.com/paper/2311.09247/) (2023)
- [Break It Down: Evidence for Structural Compositionality in Neural Networks](https://mlsanity.com/paper/2301.10884/) (2023)
- [Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models](https://mlsanity.com/paper/2502.20332/) (2025)
- [Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs](https://mlsanity.com/paper/2506.22146/) (2025)
- [Visual symbolic mechanisms: Emergent symbol processing in vision language models](https://mlsanity.com/paper/2506.15871/) (2025)
- [Object-centric Binding in Contrastive Language-Image Pretraining](https://mlsanity.com/paper/2502.14113/) (2025)
- [Whither symbols in the era of advanced neural networks?](https://mlsanity.com/paper/2508.05776/) (2025)
- [Improving Planning with Large Language Models: A Modular Agentic Architecture](https://mlsanity.com/paper/2310.00194/) (2023)
- [Evaluating Compositional Scene Understanding in Multimodal Generative Models](https://mlsanity.com/paper/2503.23125/) (2025)
- [Bound by semanticity: universal laws governing the generalization-identification tradeoff](https://mlsanity.com/paper/2506.14797/) (2025)

## Abstract (arXiv)

Despite success on standard benchmarks, vision language models display persistent failures on tasks involving processing of multi-object scenes, including many tasks that are relatively easy for humans. Recent work has found that these failures may stem from a basic inability to accurately bind object features in-context, a challenge that is referred to as the "binding problem" in cognitive science and neuroscience. The human visual system is thought to solve this binding problem via serial processing, attending to individual objects one at a time so as to avoid interference from other objects. Recent work has proposed "pointing" -- the use of explicit spatial coordinates to refer to objects -- as an analogous solution for vision language models, and found that it improves performance on challenging multi-object tasks. However, it is unclear textitwhy (i.e., on a mechanistic or representational level) this approach improves performance, and how directly this relates to serial processing in human vision. Here, we investigate this question. We find that learning to point-via-text induces an internal visual search routine, and we characterize the mechanisms that support this procedure. We also find that pointing behavior can be generalized to new tasks via fine-tuning, and that doing so eliminates binding errors and enables compositional generalization. These results provide a proof-of-principle that serial processing can solve the binding problem for vision language models just as it does for biological vision.