# Co-Me: Confidence-Guided Token Merging for Visual Geometric Transformers

arXiv: https://arxiv.org/abs/2511.14751 · HTML: https://mlsanity.com/paper/2511.14751/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving.
- Builds on 30 of its 44 references that are in the map.
- First posted on arXiv 18 Nov 2025.

## Placed in

- Efficient LLM inference and serving / [Token pruning and merging for ViTs](https://mlsanity.com/inference/token-pruning-and-merging-for-vits/)

## Builds on (most cited first, 30 of 30)

- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [Continuous 3D Perception Model with Persistent State](https://mlsanity.com/paper/2501.12387/) (2025)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass](https://mlsanity.com/paper/2501.13928/) (2025)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [3D Reconstruction with Spatial Memory](https://mlsanity.com/paper/2408.16061/) (2024)
- [Token Merging for Fast Stable Diffusion](https://mlsanity.com/paper/2303.17604/) (2023)
- [Flex Attention: A Programming Model for Generating Optimized Attention Kernels](https://mlsanity.com/paper/2412.05496/) (2024)
- [On the uncertainty of self-supervised monocular depth estimation](https://mlsanity.com/paper/2005.06209/) (2020)
- [Streaming 4D Visual Geometry Transformer](https://mlsanity.com/paper/2507.11539/) (2025)
- [MUSt3R: Multi-view Network for Stereo 3D Reconstruction](https://mlsanity.com/paper/2503.01661/) (2025)
- [TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?](https://mlsanity.com/paper/2106.11297/) (2021)
- [Token Fusion: Bridging the Gap between Token Pruning and Token Merging](https://mlsanity.com/paper/2312.01026/) (2023)
- [VGGT-Long: Chunk it, Loop it, Align it -- Pushing VGGT's Limits on Kilometer-scale Long RGB Sequences](https://mlsanity.com/paper/2507.16443/) (2025)
- [Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localization](https://mlsanity.com/paper/2412.08376/) (2024)
- [STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer](https://mlsanity.com/paper/2508.10893/) (2025)
- [Align3R: Aligned Monocular Depth Estimation for Dynamic Videos](https://mlsanity.com/paper/2412.03079/) (2024)
- [Pow3R: Empowering Unconstrained 3D Reconstruction with Camera and Scene Priors](https://mlsanity.com/paper/2503.17316/) (2025)
- [X-Pruner: eXplainable Pruning for Vision Transformers](https://mlsanity.com/paper/2303.04935/) (2023)
- [Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation](https://mlsanity.com/paper/2308.01045/) (2023)
- [SparseViT: Revisiting Activation Sparsity for Efficient High-Resolution Vision Transformer](https://mlsanity.com/paper/2303.17605/) (2023)
- [PuMer: Pruning and Merging Tokens for Efficient Vision Language Models](https://mlsanity.com/paper/2305.17530/) (2023)
- [Revisiting Token Pruning for Object Detection and Instance Segmentation](https://mlsanity.com/paper/2306.07050/) (2023)
- [A Purely Algebraic Justification of the Kabsch-Umeyama Algorithm](https://mlsanity.com/paper/1902.03138/) (2019)
- [MAC-VO: Metrics-aware Covariance for Learning-based Stereo Visual Odometry](https://mlsanity.com/paper/2409.09479/) (2024)

## Built on by (most cited first, 10 of 10)

- [Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation](https://mlsanity.com/paper/2603.23491/) (2026)
- [SAF3R: Dynamic Sparse Attention for Feed-Forward 3D Reconstruction Transformers](https://mlsanity.com/paper/2607.03612/) (2026)
- [PrePARE: Pre-AA Token Pruning for Frozen Multi-View Geometry Transformers](https://mlsanity.com/paper/2605.08371/) (2026)
- [Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers](https://mlsanity.com/paper/2605.23892/) (2026)
- [Scal3R: Learning Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction](https://mlsanity.com/paper/2609.04201/) (2026)
- [FFVO: A Feedforward Pose Decoder for Long-Horizon Visual Odometry](https://mlsanity.com/paper/2609.13733/) (2026)
- [ReSS: Residual-Restoring Sparse Attention for 3D Vision Transformers](https://mlsanity.com/paper/2609.35593/) (2026)
- [OTT3R: Multi-View 3D Reconstruction and Fast Dataset Generation at 1% Compute](https://mlsanity.com/paper/2609.36374/) (2026)
- [Does the VGGT Family Need All Its Layers?](https://mlsanity.com/paper/2609.36842/) (2026)
- [Zebra3D: Efficient Hybrid Foundation Models For Scalable 3D Reconstruction](https://mlsanity.com/paper/2610.12727/) (2026)

## Abstract (arXiv)

We propose Confidence-Guided Token Merging (Co-Me), an acceleration mechanism for visual geometric transformers without retraining or finetuning the base model. Co-Me distilled a light-weight confidence predictor to rank tokens by uncertainty and selectively merge low-confidence ones, effectively reducing computation while maintaining spatial coverage. Compared to similarity-based merging or pruning, the confidence signal in Co-Me reliably indicates regions emphasized by the transformer, enabling substantial acceleration without degrading performance. Co-Me applies seamlessly to various multi-view and streaming visual geometric transformers, achieving speedups that scale with sequence length. When applied to VGGT and Pi3, Co-Me achieves up to 21.5x and 20.4x speedup, making visual geometric transformers practical for real-time 3D perception and reconstruction.