mlsanityData to 7 Oct 2026

Can Transformers Capture Spatial Relations between Objects?

arXiv 2403.00729 ↗First posted 1 Mar 2024cs.CV cs.ROcited by 11

Not on the map. Why?

Abstract (arXiv)

Spatial relationships between objects represent key scene information for humans to understand and interact with the world. To study the capability of current computer vision systems to recognize physically grounded spatial relations, we start by proposing precise relation definitions that permit consistently annotating a benchmark dataset. Despite the apparent simplicity of this task relative to others in the recognition literature, we observe that existing approaches perform poorly on this benchmark. We propose new approaches exploiting the long-range attention capabilities of transformers for this task, and evaluating key design principles. We identify a simple "RelatiViT" architecture and demonstrate that it outperforms all current approaches. To our knowledge, this is the first method to convincingly outperform naive baselines on spatial relation prediction in in-the-wild settings. The code and datasets are available in urlhttps://sites.google.com/view/spatial-relation.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 59 of its 69 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

59 of 69 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2017Mask R-CNNcited by 7,259
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  7. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  8. 2018Link Prediction Based on Graph Neural Networkscited by 829
  9. 2021Going deeper with Image Transformerscited by 759
  10. 2024Genie: Generative Interactive Environmentscited by 706
  11. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  12. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  13. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  14. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  15. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  16. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  17. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
  18. 2016Visual Relationship Detection with Language Priorscited by 333
  19. 2018An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solutioncited by 302
  20. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 261
  21. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 258
  22. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  23. 2020GPS-Net: Graph Property Sensing Network for Scene Graph Generationcited by 165
  24. 2021Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generationcited by 137
  25. 2021Benchmarking Detection Transfer Learning with Vision Transformerscited by 130
  26. 2020Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Poolingcited by 115
  27. 2017Visual Translation Embedding Network for Visual Relation Detectioncited by 112
  28. 2017Detecting Visual Relationships with Deep Relational Networkscited by 97
  29. 2020Reasoning with Latent Structure Refinement for Document-Level Relation Extractioncited by 97
  30. 2020Learning from Context or Names? An Empirical Study on Neural Relation Extractioncited by 75
  31. 2019SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognitioncited by 55
  32. 2021SORNet: Spatial Object-Centric Representations for Sequential Manipulationcited by 48
  33. 2020Learning Visual Commonsense for Robust Scene Graph Generationcited by 47
  34. 2020HIN: Hierarchical Inference Network for Document-Level Relation Extractioncited by 42
  35. 2022Document-Level Relation Extraction with Adaptive Focal Loss and Knowledge Distillationcited by 39
  36. 2020Rel3D: A Minimally Contrastive Benchmark for Grounding Spatial Relations in 3Dcited by 30
  37. 2022RU-Net: Regularized Unrolling Network for Scene Graph Generationcited by 28
  38. 2017PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCNcited by 25

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 5.

11 papers.

  1. 2025UP-VLA: A Unified Understanding and Prediction Model for Embodied Agentcited by 99
  2. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  3. 2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 31
  4. 2025Can Multimodal Large Language Models Understand Spatial Relations?cited by 20
  5. 2025Learning 3D Object Spatial Relationships from Pre-trained 2D Diffusion Modelscited by 5
  6. 2024Believing is Seeing: Unobserved Object Detection using Generative Modelscited by 4
  7. 2026Attention in Space: Functional Roles of VLM Heads for Spatial Reasoningcited by 3
  8. 2026The Dual Mechanisms of Spatial Variable Binding in Vision-Language Modelscited by 3
  9. 2026Mitigating State Aliasing in Vision-Language-Action Models via Inverse Dynamics Learningcited by 0
  10. 2026SR-JEPA: Learning Predictive Latent State in 3D Scenescited by 0
  11. 2026Triplet2Track: A Hierarchical System with Object-Centric Representations for Reliable Long-Horizon Manipulationcited by 0