# Analyzing the Mechanism of Attention Collapse in VGGT from a Dynamics Perspective

arXiv: https://arxiv.org/abs/2512.21691 · HTML: https://mlsanity.com/paper/2512.21691/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 25 Dec 2025.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [DeepViT: Towards Deeper Vision Transformer](https://mlsanity.com/paper/2103.11886/) (2021)
- [Neighborhood Attention Transformer](https://mlsanity.com/paper/2204.07143/) (2022)
- [Stabilizing Transformer Training by Preventing Attention Entropy Collapse](https://mlsanity.com/paper/2303.06296/) (2023)
- [MUSt3R: Multi-view Network for Stereo 3D Reconstruction](https://mlsanity.com/paper/2503.01661/) (2025)
- [Scatterbrain: Unifying Sparse and Low-rank Attention Approximation](https://mlsanity.com/paper/2110.15343/) (2021)
- [SLAM3R: Real-Time Dense Scene Reconstruction from Monocular RGB Videos](https://mlsanity.com/paper/2412.09401/) (2024)
- [FastVGGT: Training-Free Acceleration of Visual Geometry Transformer](https://mlsanity.com/paper/2509.02560/) (2025)
- [WinT3R: Window-Based Streaming Reconstruction with Camera Token Pool](https://mlsanity.com/paper/2509.05296/) (2025)

## Built on by (most cited first, 6 of 6)

- [GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction](https://mlsanity.com/paper/2605.15852/) (2026)
- [VAGNet: Grounding 3D Affordance from Human-Object Interactions in Videos](https://mlsanity.com/paper/2602.20608/) (2026)
- [OmniVLA-RL: A Vision-Language-Action Model with Spatial Understanding and Online RL](https://mlsanity.com/paper/2604.17706/) (2026)
- [Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers](https://mlsanity.com/paper/2605.23892/) (2026)
- [GeoCond: A Conditioning-Aware Reliability Adapter for Feed-Forward 3D Reconstruction](https://mlsanity.com/paper/2609.18465/) (2026)
- [Stabilizing language models under continual learning via condition-anchored distillation](https://mlsanity.com/paper/2610.06940/) (2026)

## Abstract (arXiv)

Visual Geometry Grounded Transformer (VGGT) delivers state-of-the-art feed-forward 3D reconstruction, yet its global self-attention layer suffers from a drastic collapse phenomenon when the input sequence exceeds a few hundred frames: attention matrices rapidly become near rank-one, token geometry degenerates to an almost one-dimensional subspace, and reconstruction error accumulates super-linearly.In this report,we establish a rigorous mathematical explanation of the collapse by viewing the global-attention iteration as a degenerate diffusion process.We prove that,in VGGT, the token-feature flow converges toward a Dirac-type measure at a O(1/L) rate, where L is the layer index, yielding a closed-form mean-field partial differential equation that precisely predicts the empirically observed rank profile.The theory quantitatively matches the attention-heat-map evolution and a series of experiments outcomes reported in relevant works and explains why its token-merging remedy -- which periodically removes redundant tokens -- slows the effective diffusion coefficient and thereby delays collapse without additional training.We believe the analysis provides a principled lens for interpreting future scalable 3D-vision transformers,and we highlight its potential for multi-modal generalization.