# Block-Sparse Global Attention for Efficient Multi-View Geometry Transformers

arXiv: https://arxiv.org/abs/2509.07120 · HTML: https://mlsanity.com/paper/2509.07120/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 52 references that are in the map.
- First posted on arXiv 8 Sep 2025.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Continuous 3D Perception Model with Persistent State](https://mlsanity.com/paper/2501.12387/) (2025)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion](https://mlsanity.com/paper/2410.03825/) (2024)
- [Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention](https://mlsanity.com/paper/2502.11089/) (2025)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://mlsanity.com/paper/2407.02490/) (2024)
- [Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass](https://mlsanity.com/paper/2501.13928/) (2025)
- [Neural RGB-D Surface Reconstruction](https://mlsanity.com/paper/2104.04532/) (2021)
- [3D Reconstruction with Spatial Memory](https://mlsanity.com/paper/2408.16061/) (2024)
- [Neighborhood Attention Transformer](https://mlsanity.com/paper/2204.07143/) (2022)
- [Global Structure-from-Motion Revisited](https://mlsanity.com/paper/2407.20219/) (2024)
- [MoBA: Mixture of Block Attention for Long-Context LLMs](https://mlsanity.com/paper/2502.13189/) (2025)
- [FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Views](https://mlsanity.com/paper/2502.12138/) (2025)
- [MASt3R-SfM: a Fully-Integrated Solution for Unconstrained Structure-from-Motion](https://mlsanity.com/paper/2409.19152/) (2024)
- [MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Seconds](https://mlsanity.com/paper/2412.06974/) (2024)

## Built on by (most cited first, 23 of 23)

- [ZipMap: Linear-Time Stateful 3D Reconstruction via Test-Time Training](https://mlsanity.com/paper/2603.04385/) (2026)
- [VGGT-Omega](https://mlsanity.com/paper/2605.15195/) (2026)
- [VGG-T3: Offline Feed-Forward 3D Reconstruction at Scale](https://mlsanity.com/paper/2602.23361/) (2026)
- [FlashVGGT: Efficient and Scalable Visual Geometry Transformers with Compressed Descriptor Attention](https://mlsanity.com/paper/2512.01540/) (2025)
- [AVGGT: Rethinking Global Attention for Accelerating VGGT](https://mlsanity.com/paper/2512.02541/) (2025)
- [Speed3R: Sparse Feed-forward 3D Reconstruction Models](https://mlsanity.com/paper/2603.08055/) (2026)
- [HTTM: Head-wise Temporal Token Merging for Faster VGGT](https://mlsanity.com/paper/2511.21317/) (2025)
- [MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentation](https://mlsanity.com/paper/2601.06874/) (2026)
- [HeSS: Head Sensitivity Score for Sparsity Redistribution in VGGT](https://mlsanity.com/paper/2603.25336/) (2026)
- [Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective](https://mlsanity.com/paper/2604.14025/) (2026)
- [KV-Tracker: Real-Time Pose Tracking with Transformers](https://mlsanity.com/paper/2512.22581/) (2025)
- [DriveVGGT: Calibration-Constrained Visual Geometry Transformers for Multi-Camera Autonomous Driving](https://mlsanity.com/paper/2511.22264/) (2025)
- [Distill3R: A Pipeline for Democratizing 3D Foundation Models on Commodity Hardware](https://mlsanity.com/paper/2602.00865/) (2026)
- [DAGE: Dual-Stream Architecture for Efficient and Fine-Grained Geometry Estimation](https://mlsanity.com/paper/2603.03744/) (2026)
- [Tail-Aware Post-Training Quantization for 3D Geometry Models](https://mlsanity.com/paper/2602.01741/) (2026)
- [Lite3R: A Model-Agnostic Framework for Efficient Feed-Forward 3D Reconstruction](https://mlsanity.com/paper/2605.11354/) (2026)
- [TurboVGGT: Fast Visual Geometry Reconstruction with Adaptive Alternating Attention](https://mlsanity.com/paper/2605.14315/) (2026)
- [PrePARE: Pre-AA Token Pruning for Frozen Multi-View Geometry Transformers](https://mlsanity.com/paper/2605.08371/) (2026)
- [Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers](https://mlsanity.com/paper/2605.23892/) (2026)
- [ReSS: Residual-Restoring Sparse Attention for 3D Vision Transformers](https://mlsanity.com/paper/2609.35593/) (2026)
- [OTT3R: Multi-View 3D Reconstruction and Fast Dataset Generation at 1% Compute](https://mlsanity.com/paper/2609.36374/) (2026)
- [VASC: Value-Aware Sparse Attention with Cross-Layer Memory for Efficient 3D Reconstruction](https://mlsanity.com/paper/2610.01013/) (2026)
- [Zebra3D: Efficient Hybrid Foundation Models For Scalable 3D Reconstruction](https://mlsanity.com/paper/2610.12727/) (2026)

## Abstract (arXiv)

Efficient and accurate feed-forward multi-view reconstruction has long been an important task in computer vision. Recent transformer-based models like VGGT, π3 and MapAnything have demonstrated remarkable performance with relatively simple architectures. However, their scalability is fundamentally constrained by the quadratic complexity of global attention, which imposes a significant runtime bottleneck when processing large image sets. In this work, we empirically analyze the global attention matrix of these models and observe that the probability mass concentrates on a small subset of patch-patch interactions corresponding to cross-view geometric correspondences. Building on this insight and inspired by recent advances in large language models, we propose a training-free, block-sparse replacement for dense global attention, implemented with highly optimized kernels. Our method accelerates inference by more than 3× while maintaining comparable task performance. Evaluations on a comprehensive suite of multi-view benchmarks demonstrate that our approach seamlessly integrates into existing global attention-based architectures such as VGGT, π3 , and MapAnything, while substantially improving scalability to large image collections.