# The Limits of Binding in Dual Encoders

arXiv: https://arxiv.org/abs/2608.15971 · HTML: https://mlsanity.com/paper/2608.15971/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 16 Aug 2026.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Demystifying CLIP Data](https://mlsanity.com/paper/2309.16671/) (2023)
- [SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality](https://mlsanity.com/paper/2306.14610/) (2023)
- [On the Theoretical Limitations of Embedding-Based Retrieval](https://mlsanity.com/paper/2508.21038/) (2025)
- [Generalization on the Unseen, Logic Reasoning and Degree Curriculum](https://mlsanity.com/paper/2301.13105/) (2023)
- [Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models](https://mlsanity.com/paper/2305.19595/) (2023)
- [Teaching Structured Vision&Language Concepts to Vision&Language Models](https://mlsanity.com/paper/2211.11733/) (2022)
- [Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality](https://mlsanity.com/paper/2211.00768/) (2022)
- [SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations](https://mlsanity.com/paper/2406.11171/) (2024)
- [A theory of sequence indexing and working memory in recurrent neural networks](https://mlsanity.com/paper/1803.00412/) (2018)
- [A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language](https://mlsanity.com/paper/2408.12578/) (2024)
- [Discovering modular solutions that generalize compositionally](https://mlsanity.com/paper/2312.15001/) (2023)
- [Is CLIP ideal? No. Can we fix it? Yes!](https://mlsanity.com/paper/2503.08723/) (2025)
- [Bound by semanticity: universal laws governing the generalization-identification tradeoff](https://mlsanity.com/paper/2506.14797/) (2025)
- [Compositional Concept Generalization with Variational Quantum Circuits](https://mlsanity.com/paper/2509.09541/) (2025)
- [DisCoCLIP: A Distributional Compositional Tensor Network Encoder for Vision-Language Understanding](https://mlsanity.com/paper/2509.21287/) (2025)
- [Is Dimensionality a Barrier for Retrieval Models?](https://mlsanity.com/paper/2605.23556/) (2026)
- [Understanding Hardness of Vision-Language Compositionality from A Token-level Causal Lens](https://mlsanity.com/paper/2510.26302/) (2025)

## Abstract (arXiv)

Dual-encoder models such as CLIP score an image-caption pair by a single inner product of two independently computed unit vectors, and fail at binding, often scoring near chance when asked to distinguish "a red car and a blue dog" from "a blue car and a red dog". We give a mathematical account of when this failure is necessary and when it is contingent. Working within the ideal-encoder framework proposed by Kang et al., we first show the relevant axioms are satisfiable, so every impossibility must enter through an added, checkable hypothesis. We then prove three such obstructions. Depth: for recursive role-binding codes the swap margin obeys an exact law m(D) = 2b-D in the nesting depth D, with a finite-dimension version holding up to one explicitly flagged concentration estimate; the resolvable depth grows only logarithmically in the dimension and is single-digit at CLIP scale, the nesting depth of ordinary language. Objective: architecture-free throttle theorems showing that the contrastive objective's entire reward for binding is bounded by the rate at which training contrasts a caption against its own swap, a rate that vanishes at web scale, and that exactly reversed binding costs only that rate times the mean binding margin; both are verified in simulation. Geometry: a tight smoothness-binding frontier: the closer the two swap-related captions must embed to a shared paraphrase anchor, the smaller the binding margin can be, with an exact constant. Measuring its text-only diagnostic across 18 deployed text encoders, every model sits at roughly 25-35% of its ceiling, and the induced per-item ceiling tracks SugarCrepe's subset difficulty at r = 0.99. Binding failure in deployed dual encoders is thus not a dimension or smoothness limit today, but an incentive and code-structure limit, with a proved depth ceiling that remains once those are fixed.