# GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers

arXiv: https://arxiv.org/abs/2310.10375 · HTML: https://mlsanity.com/paper/2310.10375/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 84 references that are in the map.
- First posted on arXiv 16 Oct 2023.

## Builds on (most cited first, 30 of 67)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving](https://mlsanity.com/paper/2205.15997/) (2022)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images](https://mlsanity.com/paper/2206.01256/) (2022)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [Categorical Depth Distribution Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2103.01100/) (2021)

## Built on by (most cited first, 30 of 79)

- [MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images](https://mlsanity.com/paper/2403.14627/) (2024)
- [WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling](https://mlsanity.com/paper/2512.14614/) (2025)
- [Flash3D: Feed-Forward Generalisable 3D Scene Reconstruction from a Single Image](https://mlsanity.com/paper/2406.04343/) (2024)
- [MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views](https://mlsanity.com/paper/2411.04924/) (2024)
- [Cameras as Relative Positional Encoding](https://mlsanity.com/paper/2507.10496/) (2025)
- [EscherNet: A Generative Model for Scalable View Synthesis](https://mlsanity.com/paper/2402.03908/) (2024)
- [LaRa: Efficient Large-Baseline Radiance Fields](https://mlsanity.com/paper/2407.04699/) (2024)
- [Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Models](https://mlsanity.com/paper/2606.17846/) (2026)
- [Artificial Kuramoto Oscillatory Neurons](https://mlsanity.com/paper/2410.13821/) (2024)
- [3D Hand Pose Estimation in Everyday Egocentric Images](https://mlsanity.com/paper/2312.06583/) (2023)
- [What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Models](https://mlsanity.com/paper/2512.03422/) (2025)
- [Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Survey](https://mlsanity.com/paper/2507.14501/) (2025)
- [Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models](https://mlsanity.com/paper/2603.25716/) (2026)
- [PixelGaussian: Generalizable 3D Gaussian Reconstruction from Arbitrary Views](https://mlsanity.com/paper/2410.18979/) (2024)
- [Continuous Control with Coarse-to-fine Reinforcement Learning](https://mlsanity.com/paper/2407.07787/) (2024)
- [MonoSplat: Generalizable 3D Gaussian Splatting from Monocular Depth Foundation Models](https://mlsanity.com/paper/2505.15185/) (2025)
- [RayRoPE: Projective Ray Positional Encoding for Multi-view Attention](https://mlsanity.com/paper/2601.15275/) (2026)
- [Geometry-Aware Rotary Position Embedding for Consistent Video World Model](https://mlsanity.com/paper/2602.07854/) (2026)
- [ReDirector: Creating Any-Length Video Retakes with Rotary Camera Encoding](https://mlsanity.com/paper/2511.19827/) (2025)
- [Prisma-World: Camera-Controllable Multi-Agent Video World Model](https://mlsanity.com/paper/2606.09507/) (2026)
- [HouseCrafter: Lifting Floorplans to 3D Scenes with 2D Diffusion Model](https://mlsanity.com/paper/2406.20077/) (2024)
- [Scaling View Synthesis Transformers](https://mlsanity.com/paper/2602.21341/) (2026)
- [Scaling Sequence-to-Sequence Generative Neural Rendering](https://mlsanity.com/paper/2510.04236/) (2025)
- [Memorize When Needed: Decoupled Memory Control for Spatially Consistent Long-Horizon Video Generation](https://mlsanity.com/paper/2604.18215/) (2026)
- [Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction](https://mlsanity.com/paper/2411.14762/) (2024)
- [LVT: Large-Scale Scene Reconstruction via Local View Transformers](https://mlsanity.com/paper/2509.25001/) (2025)
- [URoPE: Universal Relative Position Embedding across Geometric Spaces](https://mlsanity.com/paper/2604.18747/) (2026)
- [BiWM: Advancing Open-Source Interactive Video World Models with Bidirectional Autoregression](https://mlsanity.com/paper/2606.10135/) (2026)
- [Toward Embodiment Equivariant Vision-Language-Action Policy](https://mlsanity.com/paper/2509.14630/) (2025)
- [SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformers](https://mlsanity.com/paper/2606.27345/) (2026)

## Abstract (arXiv)

As transformers are equivariant to the permutation of input tokens, encoding the positional information of tokens is necessary for many tasks. However, since existing positional encoding schemes have been initially designed for NLP tasks, their suitability for vision tasks, which typically exhibit different structural properties in their data, is questionable. We argue that existing positional encoding schemes are suboptimal for 3D vision tasks, as they do not respect their underlying 3D geometric structure. Based on this hypothesis, we propose a geometry-aware attention mechanism that encodes the geometric structure of tokens as relative transformation determined by the geometric relationship between queries and key-value pairs. By evaluating on multiple novel view synthesis (NVS) datasets in the sparse wide-baseline multi-view setting, we show that our attention, called Geometric Transform Attention (GTA), improves learning efficiency and performance of state-of-the-art transformer-based NVS models without any additional learned parameters and only minor computational overhead.