# BindCLIP: One Balanced Coupling For Compositional Vision Language Scoring

arXiv: https://arxiv.org/abs/2609.23717 · HTML: https://mlsanity.com/paper/2609.23717/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 23 references that are in the map.
- First posted on arXiv 20 Sep 2026.

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality](https://mlsanity.com/paper/2306.14610/) (2023)
- [Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem](https://mlsanity.com/paper/2411.00238/) (2024)
- [SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations](https://mlsanity.com/paper/2406.11171/) (2024)
- [Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations](https://mlsanity.com/paper/2305.06152/) (2023)
- [CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally](https://mlsanity.com/paper/2502.03566/) (2025)
- [Explain Before You Answer: A Survey on Compositional Visual Reasoning](https://mlsanity.com/paper/2508.17298/) (2025)
- [ComCLIP: Training-Free Compositional Image and Text Matching](https://mlsanity.com/paper/2211.13854/) (2022)
- [The Hard Positive Truth about Vision-Language Compositionality](https://mlsanity.com/paper/2409.17958/) (2024)
- [ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs](https://mlsanity.com/paper/2406.08164/) (2024)
- [Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs](https://mlsanity.com/paper/2506.22146/) (2025)
- [Vision-Language Models Struggle to Align Entities across Modalities](https://mlsanity.com/paper/2503.03854/) (2025)
- [Object-centric Binding in Contrastive Language-Image Pretraining](https://mlsanity.com/paper/2502.14113/) (2025)
- [Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data](https://mlsanity.com/paper/2503.01167/) (2025)
- [Seeing What Matters: Empowering CLIP with Patch Generation-to-Selection](https://mlsanity.com/paper/2503.17080/) (2025)
- [Common Data Properties Limit Object-Attribute Binding in CLIP](https://mlsanity.com/paper/2507.07985/) (2025)
- [Revisiting Compositionality in Dual-Encoder Vision-Language Models: The Role of Inference](https://mlsanity.com/paper/2604.11496/) (2026)

## Abstract (arXiv)

Global vision--language similarities compress an image and a caption into one vector, preserving semantics but not which word corresponds to which region or how those regions are arranged; a model can recognize every word and object yet prefer a compositionally incorrect caption. We argue that a frozen encoder retains this association structure, so the problem is to read it rather than to rebuild it beside the pretrained similarity. We introduce BindCLIP, a pairwise scorer built on one latent object: a balanced token--patch--depth optimal-transport coupling that places both candidate captions and several visual depths in a single plan. Semantic, entity, order, and spatial evidence are read as energies of this state, and exchanging the candidates permutes the plan, making the score exactly antisymmetric. A geometric refinement inside the coupling contracts moves that the candidates and the visual depths do not support. No task label, parser, relation inventory, or detector is used. One checkpoint and one inference path improve the official What'sUp, ARO, and SugarCrepe benchmarks over frozen global CLIP, with the strongest transfer on the relation splits. Controls rule out patch access and caption-length shortcuts, and an inference-time lesion localizes spatial arrangement to the coupling.