mlsanityData to 7 Oct 2026

GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers

arXiv 2310.10375 ↗First posted 16 Oct 2023cs.CV cs.AI cs.LGcited by 75

Not on the map. Why?

Abstract (arXiv)

As transformers are equivariant to the permutation of input tokens, encoding the positional information of tokens is necessary for many tasks. However, since existing positional encoding schemes have been initially designed for NLP tasks, their suitability for vision tasks, which typically exhibit different structural properties in their data, is questionable. We argue that existing positional encoding schemes are suboptimal for 3D vision tasks, as they do not respect their underlying 3D geometric structure. Based on this hypothesis, we propose a geometry-aware attention mechanism that encodes the geometric structure of tokens as relative transformation determined by the geometric relationship between queries and key-value pairs. By evaluating on multiple novel view synthesis (NVS) datasets in the sparse wide-baseline multi-view setting, we show that our attention, called Geometric Transform Attention (GTA), improves learning efficiency and performance of state-of-the-art transformer-based NVS models without any additional learned parameters and only minor computational overhead.

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 67 of its 84 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

67 of 84 references.

References · 67

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  10. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  11. 2017Non-local Neural Networkscited by 2,290
  12. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
38 more
  1. 2021Vision Transformers for Dense Predictioncited by 1,789
  2. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  3. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  4. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  5. 2019Root Mean Square Layer Normalizationcited by 1,425
  6. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  9. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  10. 2017Dynamic Routing Between Capsulescited by 689
  11. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  12. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  13. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  14. 2022TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Drivingcited by 477
  15. 2022Kubric: A scalable dataset generatorcited by 373
  16. 2022PETRv2: A Unified Framework for 3D Perception from Multi-Camera Imagescited by 351
  17. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
  18. 2021Categorical Depth Distribution Network for Monocular 3D Object Detectioncited by 326
  19. 2023Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detectioncited by 313
  20. 2022Novel View Synthesis with Diffusion Modelscited by 299
  21. 2022Unifying Flow, Stereo and Depth Estimationcited by 270
  22. 2022Cross-view Transformers for real-time Map-view Semantic Segmentationcited by 264
  23. 2019Gauge Equivariant Convolutional Networks and the Icosahedral CNNcited by 262
  24. 2020Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Imagecited by 239
  25. 2021Geometric and Physical Quantities Improve E(3) Equivariant Message Passingcited by 225
  26. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 222
  27. 2022Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformercited by 211
  28. 2018Stereo Magnification: Learning View Synthesis using Multiplane Imagescited by 206
  29. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
  30. 2022Geometric Transformer for Fast and Robust Point Cloud Registrationcited by 200
  31. 2021GeoNeRF: Generalizing NeRF with Geometry Priorscited by 195
  32. 2021Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenescited by 190
  33. 2022BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigscited by 123
  34. 2022A Length-Extrapolatable Transformercited by 123
  35. 2022Generalizable Patch-Based Neural Renderingcited by 121
  36. 2020Epipolar Transformerscited by 117
  37. 2021Translating Images into Mapscited by 115
  38. 2022Is Attention All That NeRF Needs?cited by 113

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 10; 2025: 17; 2026: 46.

75 papers.

Sort:

By area:

  1. 2024MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Imagescited by 553
  2. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 152
  3. 2024Flash3D: Feed-Forward Generalisable 3D Scene Reconstruction from a Single Imagecited by 113
  4. 2024MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Viewscited by 111
  5. 2024EscherNet: A Generative Model for Scalable View Synthesiscited by 94
  6. 2025Cameras as Relative Positional Encodingcited by 94
  7. 2024LaRa: Efficient Large-Baseline Radiance Fieldscited by 54
  8. 2026Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Modelscited by 47
  9. 2024Artificial Kuramoto Oscillatory Neuronscited by 37
  10. 20233D Hand Pose Estimation in Everyday Egocentric Imagescited by 33
  11. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  12. 2025Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Surveycited by 23
38 more
  1. 2026Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Modelscited by 20
  2. 2024PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Viewscited by 18
  3. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  4. 2025MonoSplat: Generalizable 3D Gaussian Splatting from Monocular Depth Foundation Modelscited by 15
  5. 2026Geometry-Aware Rotary Position Embedding for Consistent Video World Modelcited by 12
  6. 2025ReDirector: Creating Any-Length Video Retakes with Rotary Camera Encodingcited by 10
  7. 2026RayRoPE: Projective Ray Positional Encoding for Multi-view Attentioncited by 10
  8. 2024HouseCrafter: Lifting Floorplans to 3D Scenes with 2D Diffusion Modelcited by 9
  9. 2025Scaling Sequence-to-Sequence Generative Neural Renderingcited by 8
  10. 2026Prisma-World: Camera-Controllable Multi-Agent Video World Modelcited by 8
  11. 2024Efficient Long Video Tokenization via Coordinate-based Patch Reconstructioncited by 6
  12. 2025LVT: Large-Scale Scene Reconstruction via Local View Transformerscited by 6
  13. 2026Memorize When Needed: Decoupled Memory Control for Spatially Consistent Long-Horizon Video Generationcited by 6
  14. 2025Toward Embodiment Equivariant Vision-Language-Action Policycited by 5
  15. 2026Scaling View Synthesis Transformerscited by 5
  16. 2026URoPE: Universal Relative Position Embedding across Geometric Spacescited by 5
  17. 2026BiWM: Advancing Open-Source Interactive Video World Models with Bidirectional Autoregressioncited by 5
  18. 2026Context-Matched Distillation: Teacher Causality for Autoregressive Video Distillationcited by 5
  19. 2023Mitigating Perspective Distortion-induced Shape Ambiguity in Image Cropscited by 4
  20. 2025DriveVGGT: Calibration-Constrained Visual Geometry Transformers for Multi-Camera Autonomous Drivingcited by 4
  21. 2026Kuramoto Oscillatory Phase Encoding: Neuro-inspired Synchronization for Improved Learning Efficiencycited by 3
  22. 2026CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generationcited by 3
  23. 2026SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformerscited by 3
  24. 2026DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulationcited by 3
  25. 2025High Dynamic Range Novel View Synthesis with Single Exposurecited by 2
  26. 2025HUMOF: Human Motion Forecasting in Interactive Social Scenescited by 2
  27. 2025GimbalDiffusion: Gravity-Aware Camera Control for Video Generationcited by 2
  28. 2025StereoSpace: Depth-Free Synthesis of Stereo Geometry via End-to-End Diffusion in a Canonical Spacecited by 2
  29. 2026UniScale: Unified Scale-Aware 3D Reconstruction for Multi-View Understanding via Prior Injection for Robotic Perceptioncited by 2
  30. 2026RnG: A Unified Transformer for Complete 3D Modeling from Partial Observationscited by 2
  31. 2026SemGS: Feed-Forward Semantic 3D Gaussian Splatting from Sparse Views for Generalizable Scene Understandingcited by 2
  32. 2026R3DP: Real-Time 3D-Aware Policy for Embodied Manipulationcited by 2
  33. 2026Stereo World Model: Camera-Guided Stereo Video Generationcited by 2
  34. 2026Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long Generalizationcited by 2
  35. 2026DecMem: Towards Minute-Long Consistent World Generation with Decoupled Memorycited by 2
  36. 2026RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathwayscited by 2
  37. 2025Inter-environmental world modeling for continuous and compositional dynamicscited by 1
  38. 20262Xplat: Decoupling Geometry and Appearance Modeling for Feed-Forward 3D Gaussian Splattingcited by 1

50 most cited of 75 All 75 (JSON)