mlsanityData to 14 Oct 2026

Geometry without Position? When Positional Embeddings Help and Hurt Spatial Reasoning

arXiv 2601.22231 ↗First posted 29 Jan 2026cs.CVcited by 0

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

This paper revisits the role of positional embeddings (PEs) within vision transformers (ViTs) from a geometric perspective. We show that PEs are not mere token indices but effectively function as geometric priors that shape the spatial structure of the representation. We introduce token-level diagnostics that measure how multi-view geometric consistency in ViT representation depends on consitent PEs. Through extensive experiments on 14 foundation ViT models, we reveal how PEs influence multi-view geometry and spatial reasoning. Our findings clarify the role of PEs as a causal mechanism that governs spatial structure in ViT representations. Our code is provided in https://github.com/shijianjian/vit-geometry-probes

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 46 of its 52 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

46 of 52 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2023Segment Anythingcited by 8,234
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
34 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  2. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  3. 2025DINOv3cited by 1,414
  4. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,360
  5. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  6. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  7. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  8. 2018Self-Attention with Relative Position Representationscited by 1,120
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  10. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
  11. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  12. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 984
  13. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 968
  14. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  15. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  16. 2018Pyramid Stereo Matching Networkcited by 489
  17. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  18. 2021Conditional Positional Encodings for Vision Transformerscited by 406
  19. 2017End-to-End Learning of Geometry and Context for Deep Stereo Regressioncited by 360
  20. 2021Deep ViT Features as Dense Visual Descriptorscited by 330
  21. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 207
  22. 2020Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformerscited by 190
  23. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 177
  24. 2023Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereocited by 172
  25. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  26. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 127
  27. 2019DPSNet: End-to-end Deep Plane Sweep Stereocited by 115
  28. 2025Cameras as Relative Positional Encodingcited by 105
  29. 2025AnyUp: Universal Feature Upsamplingcited by 28
  30. 2024Feat2GS: Probing Visual Foundation Models with Gaussian Splattingcited by 27
  31. 2025Positional Encoding Fieldcited by 20
  32. 2023When Epipolar Constraint Meets Non-local Operators in Multi-View Stereocited by 14
  33. 2024Multi-label Cluster Discrimination for Visual Representation Learningcited by 14
  34. 2025Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddingscited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Intrinsic Limits of Transformer Image Embeddings in Non-Solvable Spatial Reasoning
  2. 2026How Vision Transformers Represent Position: A Mechanistic Comparison of Absolute and Rotary Positional Encodings
  3. 2026GeoWeaver: Grounding Visual Tokens with Geometric Evidence before Scene Reasoning
  4. 2026Make Geometry Matter for Spatial Reasoning
  5. 2024Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects
  6. 2026Why Far Looks Up: Probing Spatial Representation in Vision-Language Models
  7. 2026What Makes a Good Visual Representation for Spatial Understanding in MLLMs?
  8. 2026Semantic Richness or Geometric Reasoning? The Fragility of VLM's Visual Invariance