mlsanityData to 7 Oct 2026

What's left can't be right -- The remaining positional incompetence of contrastive vision-language models

arXiv 2311.11477 ↗First posted 20 Nov 2023cs.CV cs.CLcited by 5

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Contrastive vision-language models like CLIP have been found to lack spatial understanding capabilities. In this paper we discuss the possible causes of this phenomenon by analysing both datasets and embedding space. By focusing on simple left-right positional relations, we show that this behaviour is entirely predictable, even with large-scale datasets, demonstrate that these relations can be taught using synthetic data and show that this approach can generalise well to natural images - improving the performance on left-right relations on Visual Genome Relations.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 34 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

20 of 34 references.

References · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  10. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  11. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  12. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
8 more
  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  2. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  3. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  4. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  5. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  6. 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 75
  7. 2021How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropycited by 23
  8. 2021TraVLR: Now You See It, Now You Don't! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2024MARVEL-40M+: Multi-Level Visual Elaboration for High-Fidelity Text-to-3D Content Creationcited by 7
  2. 2025Metamorphic Testing for Pose Estimation Systemscited by 3
  3. 2025Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Taskscited by 1
  4. 2025Dynamic Relation Inference via Verb Embeddingscited by 0
  5. 2026Seizure-Semiology-Suite (S3): A Clinically Multimodal Dataset, Benchmark, and Models for Seizure Semiology Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Left-Right Symmetry Breaking in CLIP-style Vision-Language Models Trained on Synthetic Spatial-Relation Data
  2. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning
  3. 2026The Spatial Blindspot of Vision-Language Models
  4. 2026Why Far Looks Up: Probing Spatial Representation in Vision-Language Models
  5. 2025Investigating Spatial Attention Bias in Vision-Language Models
  6. 2026Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA
  7. 2026Auto-Comp: An Automated Pipeline for Scalable Compositional Probing of Contrastive Vision-Language Models
  8. 2022Visual Spatial Reasoning