# Fusion: A Framework for Unified Sequential Token AdaptatIon in VisiOn TraNsformers

arXiv: https://arxiv.org/abs/2607.02612 · HTML: https://mlsanity.com/paper/2607.02612/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving.
- Builds on 19 of its 23 references that are in the map.
- First posted on arXiv 1 Jul 2026.

## Placed in

- Efficient LLM inference and serving / [Token pruning and merging for ViTs](https://mlsanity.com/inference/token-pruning-and-merging-for-vits/)

## Builds on (most cited first, 19 of 19)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [Slimmable Neural Networks](https://mlsanity.com/paper/1812.08928/) (2018)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [Adaptive Token Sampling For Efficient Vision Transformers](https://mlsanity.com/paper/2111.15667/) (2021)
- [SPViT: Enabling Faster Vision Transformers via Soft Token Pruning](https://mlsanity.com/paper/2112.13890/) (2021)
- [Token Fusion: Bridging the Gap between Token Pruning and Token Merging](https://mlsanity.com/paper/2312.01026/) (2023)
- [CF-ViT: A General Coarse-to-Fine Method for Vision Transformer](https://mlsanity.com/paper/2203.03821/) (2022)
- [Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentation](https://mlsanity.com/paper/2308.01045/) (2023)
- [LGViT: Dynamic Early Exiting for Accelerating Vision Transformer](https://mlsanity.com/paper/2308.00255/) (2023)
- [Learned Thresholds Token Merging and Pruning for Vision Transformers](https://mlsanity.com/paper/2307.10780/) (2023)
- [SaiT: Sparse Vision Transformers through Adaptive Token Pruning](https://mlsanity.com/paper/2210.05832/) (2022)

## Built on by (most cited first, 2 of 2)

- [ProgResViT: Progressive Resolution and Width for Adaptive Vision Transformers](https://mlsanity.com/paper/2609.03216/) (2026)
- [REQAP: Resilient Weight Packing and Quantization for Edge DNN Acceleration](https://mlsanity.com/paper/2609.17555/) (2026)

## Abstract (arXiv)

Vision Transformers achieve strong image classification accuracy but process all image regions with nearly the same computation, even when many regions are redundant or uninformative. Recent adaptive inference methods reduce this cost by selectively compressing tokens or terminating inference early, but combining these mechanisms often causes unstable intermediate representations and accuracy degradation. We introduce Fusion, a unified adaptive inference framework that coordinates token merging, early exiting, and token pruning through a simple staged design: tokens are merged first, confidence is evaluated next, and pruning is applied only to samples that continue inference. This ordering allows the three mechanisms to operate cooperatively rather than competitively. Fusion further includes lightweight routing modules that adapt compression strength to each input and support inference-time adjustment of the accuracy--latency trade-off without retraining. On ImageNet-1k with DeiT-S, Fusion matches or surpasses state-of-the-art adaptive ViT methods at comparable compute budgets while reducing calibration error by up to 4× and inference energy by 48%. Experiments across ImageNet-100, CIFAR-100, and ImageNette with multiple ViT backbones demonstrate consistent transferability without dataset-specific tuning.