# AVGGT: Rethinking Global Attention for Accelerating VGGT

arXiv: https://arxiv.org/abs/2512.02541 · HTML: https://mlsanity.com/paper/2512.02541/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 2 Dec 2025.

## Builds on (most cited first, 22 of 22)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [MapAnything: Universal Feed-Forward Metric 3D Reconstruction](https://mlsanity.com/paper/2509.13414/) (2025)
- [Neural RGB-D Surface Reconstruction](https://mlsanity.com/paper/2104.04532/) (2021)
- [Global Structure-from-Motion Revisited](https://mlsanity.com/paper/2407.20219/) (2024)
- [Pixel-Perfect Structure-from-Motion with Featuremetric Refinement](https://mlsanity.com/paper/2108.08291/) (2021)
- [CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion](https://mlsanity.com/paper/2210.10716/) (2022)
- [SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference](https://mlsanity.com/paper/2502.18137/) (2025)
- [VGGT-Long: Chunk it, Loop it, Align it -- Pushing VGGT's Limits on Kilometer-scale Long RGB Sequences](https://mlsanity.com/paper/2507.16443/) (2025)
- [TransformerFusion: Monocular RGB Scene Reconstruction using Transformers](https://mlsanity.com/paper/2107.02191/) (2021)
- [Not All Tokens Are Equal: Human-centric Visual Analysis via Token Clustering Transformer](https://mlsanity.com/paper/2204.08680/) (2022)
- [FastVGGT: Training-Free Acceleration of Visual Geometry Transformer](https://mlsanity.com/paper/2509.02560/) (2025)
- [Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models](https://mlsanity.com/paper/2112.00029/) (2021)
- [Learning to Merge Tokens in Vision Transformers](https://mlsanity.com/paper/2202.12015/) (2022)
- [Which Tokens to Use? Investigating Token Reduction in Vision Transformers](https://mlsanity.com/paper/2308.04657/) (2023)
- [Block-Sparse Global Attention for Efficient Multi-View Geometry Transformers](https://mlsanity.com/paper/2509.07120/) (2025)

## Built on by (most cited first, 16 of 16)

- [MeMix: Writing Less, Remembering More for Streaming 3D Reconstruction](https://mlsanity.com/paper/2603.15330/) (2026)
- [Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective](https://mlsanity.com/paper/2604.14025/) (2026)
- [Distill3R: A Pipeline for Democratizing 3D Foundation Models on Commodity Hardware](https://mlsanity.com/paper/2602.00865/) (2026)
- [TurboVGGT: Fast Visual Geometry Reconstruction with Adaptive Alternating Attention](https://mlsanity.com/paper/2605.14315/) (2026)
- [RegimeVGGT: Layer-Wise Spatially Preserving Redundancy Removal for Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2606.18439/) (2026)
- [PRISM: Feed-Forward Single-Image 3D Reconstruction via Geometric Warp-Residual Modeling](https://mlsanity.com/paper/2606.25430/) (2026)
- [Diversity-aware View Partitioning for Scalable VGGT](https://mlsanity.com/paper/2607.01885/) (2026)
- [SAF3R: Dynamic Sparse Attention for Feed-Forward 3D Reconstruction Transformers](https://mlsanity.com/paper/2607.03612/) (2026)
- [ReCoSplat: Online Feed-Forward Gaussian Splatting via Render-and-Compare](https://mlsanity.com/paper/2603.09968/) (2026)
- [Not All Tasks Quantize Equally: Fisher-Guided Quantization for Visual Geometry Transformer](https://mlsanity.com/paper/2605.15828/) (2026)
- [Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers](https://mlsanity.com/paper/2605.23892/) (2026)
- [Scal3R: Learning Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction](https://mlsanity.com/paper/2609.04201/) (2026)
- [OTT3R: Multi-View 3D Reconstruction and Fast Dataset Generation at 1% Compute](https://mlsanity.com/paper/2609.36374/) (2026)
- [Does the VGGT Family Need All Its Layers?](https://mlsanity.com/paper/2609.36842/) (2026)
- [VASC: Value-Aware Sparse Attention with Cross-Layer Memory for Efficient 3D Reconstruction](https://mlsanity.com/paper/2610.01013/) (2026)
- [Zebra3D: Efficient Hybrid Foundation Models For Scalable 3D Reconstruction](https://mlsanity.com/paper/2610.12727/) (2026)

## Abstract (arXiv)

Models such as VGGT and π3 have shown strong multi-view 3D performance, but their heavy reliance on global self-attention results in high computational cost. Existing sparse-attention variants offer partial speedups, yet lack a systematic analysis of how global attention contributes to multi-view reasoning. In this paper, we first conduct an in-depth investigation of the global attention modules in VGGT and π3 to better understand their roles. Our analysis reveals a clear division of roles in the alternating global-frame architecture: early global layers do not form meaningful correspondences, middle layers perform cross-view alignment, and last layers provide only minor refinements. Guided by these findings, we propose a training-free two-step acceleration scheme: (1) converting early global layers into frame attention, and (2) subsampling global attention by subsampling K/V over patch tokens with diagonal preservation and a mean-fill component. We instantiate this strategy on VGGT and π3 and evaluate across standard pose and point-map benchmarks. Our method achieves substantial inference acceleration across different context lengths, yielding about 2× speedup at 100 frames, 4--5× at 300 frames, and 8--10× at 800 frames, while matching or slightly improving the accuracy of the original models and remaining robust in extremely dense multi-view settings where prior sparse-attention baselines fail.