Does the VGGT Family Need All Its Layers?
Not on the map. Why?
Abstract (arXiv)
Which layers of a feed-forward geometry model are needed to preserve both camera poses and dense 3D structure? We study layer redundancy in VGGT, π3, and VGGT-Omega: 3,018 pruned configurations, scored on seven camera-pose and dense-geometry metrics across four indoor and outdoor datasets. Four findings follow: (i) Removable layers cluster in two redundancy regions: a dominant early region and a narrower late one, while deletions spanning the intervening layers are consistently more disruptive. This recurring pattern holds across models, datasets, and metrics, and contrasts with the middle-to-late redundancy commonly reported in the literature. (ii) Within these regions, we observe that the joint degradation from deleting two intervals is approximately the sum of their individual degradations, reducing the number of model evaluations for pruning search from O(L4) to O(L2), where L is the aggregator depth. (iii) We find that CKA provides a cheaper representation-based proxy for interval degradation, offering a practical trade-off between pruning quality and calibration cost. (iv) Closed-form linear calibration recovers accuracy after pruning without end-to-end retraining. A least-squares analysis shows that using a shared map for special and patch tokens generally incurs excess reconstruction loss, motivating token-aware recovery. Recovery maps fitted on just 100 calibration scenes generalize to held-out scenes and unseen datasets. The resulting models reduce aggregator parameters by up to 44% while maintaining accuracy comparable to their intact counterparts. Code and experimental results will be available at our project page: https://xian-bei.github.io/vggt-family-layer-redundancy/
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 53 of its 55 references that are in the map. First posted on arXiv 29 Sep 2026.
Builds on
53 of 55 references.
References · 53
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
- 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2023Qwen Technical Reportcited by 3,344
- 2023Mistral 7Bcited by 3,164
- 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
- 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
38 more
- 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
- 2019Similarity of Neural Network Representations Revisitedcited by 1,755
- 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
- 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
- 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
- 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
- 2024Grounding Image Matching in 3D with MASt3Rcited by 983
- 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
- 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
- 2024MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motioncited by 442
- 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
- 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 367
- 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
- 2025π3: Permutation-Equivariant Visual Geometry Learningcited by 319
- 2022How Do Vision Transformers Work?cited by 307
- 2021DeepViT: Towards Deeper Vision Transformercited by 289
- 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
- 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
- 2024MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Secondscited by 141
- 2025MUSt3R: Multi-view Network for Stereo 3D Reconstructioncited by 141
- 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 126
- 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
- 2025FastVGGT: Training-Free Acceleration of Visual Geometry Transformercited by 98
- 2024What Matters in Transformers? Not All Attention is Neededcited by 84
- 2025The Curse of Depth in Large Language Modelscited by 62
- 2022Teaching Matters: Investigating the Role of Supervision in Vision Transformerscited by 45
- 2024Streamlining Redundant Layers to Compress Large Language Modelscited by 35
- 2025Quantized Visual Geometry Grounded Transformercited by 17
- 2025LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Mergingcited by 17
- 2025AVGGT: Rethinking Global Attention for Accelerating VGGTcited by 16
- 2025ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearizationcited by 13
- 2025Co-Me: Confidence-Guided Token Merging for Visual Geometric Transformerscited by 10
- 2025HTTM: Head-wise Temporal Token Merging for Faster VGGTcited by 9
- 2025A Simple Linear Patch Revives Layer-Pruned Large Language Modelscited by 4
- 2026Rethinking Layer Relevance in Large Language Models Beyond Cosine Similaritycited by 4
- 2026SIMPLER: Efficient Foundation Model Adaptation via Similarity-Guided Layer Pruning for Earth Observationcited by 2
- 2026Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruningcited by 2
- 2026Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMscited by 2
50 most cited of 53 All 53 (JSON)
Built on by
Not cited yet.