mlsanityData to 14 Oct 2026

Does the VGGT Family Need All Its Layers?

arXiv 2609.36842 ↗First posted 29 Sep 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Which layers of a feed-forward geometry model are needed to preserve both camera poses and dense 3D structure? We study layer redundancy in VGGT, π3, and VGGT-Omega: 3,018 pruned configurations, scored on seven camera-pose and dense-geometry metrics across four indoor and outdoor datasets. Four findings follow: (i) Removable layers cluster in two redundancy regions: a dominant early region and a narrower late one, while deletions spanning the intervening layers are consistently more disruptive. This recurring pattern holds across models, datasets, and metrics, and contrasts with the middle-to-late redundancy commonly reported in the literature. (ii) Within these regions, we observe that the joint degradation from deleting two intervals is approximately the sum of their individual degradations, reducing the number of model evaluations for pruning search from O(L4) to O(L2), where L is the aggregator depth. (iii) We find that CKA provides a cheaper representation-based proxy for interval degradation, offering a practical trade-off between pruning quality and calibration cost. (iv) Closed-form linear calibration recovers accuracy after pruning without end-to-end retraining. A least-squares analysis shows that using a shared map for special and patch tokens generally incurs excess reconstruction loss, motivating token-aware recovery. Recovery maps fitted on just 100 calibration scenes generalize to held-out scenes and unseen datasets. The resulting models reduce aggregator parameters by up to 44% while maintaining accuracy comparable to their intact counterparts. Code and experimental results will be available at our project page: https://xian-bei.github.io/vggt-family-layer-redundancy/

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 53 of its 55 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

53 of 55 references.

References · 53

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2023Mistral 7Bcited by 3,164
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  12. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
38 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  2. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  3. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  4. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
  5. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
  6. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  7. 2024Grounding Image Matching in 3D with MASt3Rcited by 983
  8. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  9. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  10. 2024MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motioncited by 442
  11. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  12. 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 367
  13. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  14. 2025π3: Permutation-Equivariant Visual Geometry Learningcited by 319
  15. 2022How Do Vision Transformers Work?cited by 307
  16. 2021DeepViT: Towards Deeper Vision Transformercited by 289
  17. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  18. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  19. 2024MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Secondscited by 141
  20. 2025MUSt3R: Multi-view Network for Stereo 3D Reconstructioncited by 141
  21. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 126
  22. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  23. 2025FastVGGT: Training-Free Acceleration of Visual Geometry Transformercited by 98
  24. 2024What Matters in Transformers? Not All Attention is Neededcited by 84
  25. 2025The Curse of Depth in Large Language Modelscited by 62
  26. 2022Teaching Matters: Investigating the Role of Supervision in Vision Transformerscited by 45
  27. 2024Streamlining Redundant Layers to Compress Large Language Modelscited by 35
  28. 2025Quantized Visual Geometry Grounded Transformercited by 17
  29. 2025LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Mergingcited by 17
  30. 2025AVGGT: Rethinking Global Attention for Accelerating VGGTcited by 16
  31. 2025ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearizationcited by 13
  32. 2025Co-Me: Confidence-Guided Token Merging for Visual Geometric Transformerscited by 10
  33. 2025HTTM: Head-wise Temporal Token Merging for Faster VGGTcited by 9
  34. 2025A Simple Linear Patch Revives Layer-Pruned Large Language Modelscited by 4
  35. 2026Rethinking Layer Relevance in Large Language Models Beyond Cosine Similaritycited by 4
  36. 2026SIMPLER: Efficient Foundation Model Adaptation via Similarity-Guided Layer Pruning for Earth Observationcited by 2
  37. 2026Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruningcited by 2
  38. 2026Ghosted Layers: Unconstrained Activation Alignment for Recovering Layer-Pruned LLMscited by 2

50 most cited of 53 All 53 (JSON)

Built on by

Not cited yet.