# Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers

arXiv: https://arxiv.org/abs/2605.23892 · HTML: https://mlsanity.com/paper/2605.23892/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 104 of its 115 references that are in the map.
- First posted on arXiv 22 May 2026.

## Builds on (most cited first, 30 of 104)

- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass](https://mlsanity.com/paper/2501.13928/) (2025)
- [Learning to (Learn at Test Time): RNNs with Expressive Hidden States](https://mlsanity.com/paper/2407.04620/) (2024)
- [π3: Permutation-Equivariant Visual Geometry Learning](https://mlsanity.com/paper/2507.13347/) (2025)
- [MapAnything: Universal Feed-Forward Metric 3D Reconstruction](https://mlsanity.com/paper/2509.13414/) (2025)
- [Neural RGB-D Surface Reconstruction](https://mlsanity.com/paper/2104.04532/) (2021)
- [ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residuals](https://mlsanity.com/paper/1905.02082/) (2019)
- [FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Views](https://mlsanity.com/paper/2502.12138/) (2025)
- [CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow](https://mlsanity.com/paper/2211.10408/) (2022)
- [MASt3R-SfM: a Fully-Integrated Solution for Unconstrained Structure-from-Motion](https://mlsanity.com/paper/2409.19152/) (2024)
- [Streaming 4D Visual Geometry Transformer](https://mlsanity.com/paper/2507.11539/) (2025)
- [CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion](https://mlsanity.com/paper/2210.10716/) (2022)
- [MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Seconds](https://mlsanity.com/paper/2412.06974/) (2024)
- [MUSt3R: Multi-view Network for Stereo 3D Reconstruction](https://mlsanity.com/paper/2503.01661/) (2025)
- [TTT3R: 3D Reconstruction as Test-Time Training](https://mlsanity.com/paper/2509.26645/) (2025)
- [VGGT-Long: Chunk it, Loop it, Align it -- Pushing VGGT's Limits on Kilometer-scale Long RGB Sequences](https://mlsanity.com/paper/2507.16443/) (2025)
- [Self-Attention Networks Can Process Bounded Hierarchical Languages](https://mlsanity.com/paper/2105.11115/) (2021)
- [Easi3R: Estimating Disentangled Motion from DUSt3R Without Training](https://mlsanity.com/paper/2503.24391/) (2025)
- [STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer](https://mlsanity.com/paper/2508.10893/) (2025)
- [Align3R: Aligned Monocular Depth Estimation for Dynamic Videos](https://mlsanity.com/paper/2412.03079/) (2024)
- [Pow3R: Empowering Unconstrained 3D Reconstruction with Camera and Scene Priors](https://mlsanity.com/paper/2503.17316/) (2025)
- [Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstruction](https://mlsanity.com/paper/2504.07961/) (2025)
- [MegaLoc: One Retrieval to Place Them All](https://mlsanity.com/paper/2502.17237/) (2025)
- [InfiniteVGGT: Visual Geometry Grounded Transformer for Endless Streams](https://mlsanity.com/paper/2601.02281/) (2026)
- [Dens3R: A Foundation Model for 3D Geometry Prediction](https://mlsanity.com/paper/2507.16290/) (2025)
- [LONG3R: Long Sequence Streaming 3D Reconstruction](https://mlsanity.com/paper/2507.18255/) (2025)
- [Dynamic Point Maps: A Versatile Representation for Dynamic 3D Reconstruction](https://mlsanity.com/paper/2503.16318/) (2025)

## Abstract (arXiv)

Visual geometry transformers have become powerful architectures for multi-view 3D reconstruction, enabling joint prediction of multiple 3D attributes in a feed-forward manner. However, their computational cost grows quadratically with the input sequence length due to the global attention layers inside these models. This limits both their scalability and efficiency. In this work, we address this challenge with a simple yet general strategy: restricting the number of key/value tokens that each query interacts with during global attention. To achieve effective token selection, we introduce a two-stage framework. First, an inter-frame selection step operates at the frame level to identify frames that should be preserved. Second, an intra-frame selection step further discards more redundant tokens within the selected frames. Our analysis highlights the advantage of a diversity-based strategy for inter-frame selection, which ensures broad coverage of the scene. For intra-frame selection, we show that layer-aware sparsification is necessary, with the selection process guided by the entropy of the global attention pattern. Our approach offers a superior speed-accuracy trade-off compared to existing solutions. Extensive experiments show that it accelerates visual geometry transformers by over 85% for scenes with 500 images while maintaining, or even improving, baseline performance, which hints that how our token selection strategy can play a crucial role in future applications of visual geometry transformers. Our project website is available at https://zsh2000.github.io/good-token-hunting.github.io.