mlsanityData to 7 Oct 2026

GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagation

arXiv 2311.03035 ↗First posted 6 Nov 2023cs.CV cs.AIcited by 19

On the map: Efficient LLM inference and serving → Token pruning and merging for ViTs

Abstract (arXiv)

Vision Transformers (ViTs) have revolutionized the field of computer vision, yet their deployments on resource-constrained devices remain challenging due to high computational demands. To expedite pre-trained ViTs, token pruning and token merging approaches have been developed, which aim at reducing the number of tokens involved in the computation. However, these methods still have some limitations, such as image information loss from pruned tokens and inefficiency in the token-matching process. In this paper, we introduce a novel Graph-based Token Propagation (GTP) method to resolve the challenge of balancing model efficiency and information preservation for efficient ViTs. Inspired by graph summarization algorithms, GTP meticulously propagates less significant tokens' information to spatially and semantically connected tokens that are of greater importance. Consequently, the remaining few tokens serve as a summarization of the entire token graph, allowing the method to reduce computational complexity while preserving essential information of eliminated tokens. Combined with an innovative token selection strategy, GTP can efficiently identify image tokens to be propagated. Extensive experiments have validated GTP's effectiveness, demonstrating both efficiency and performance improvements. Specifically, GTP decreases the computational complexity of both DeiT-S and DeiT-B by up to 26% with only a minimal 0.3% accuracy drop on ImageNet-1K without finetuning, and remarkably surpasses the state-of-the-art token merging method on various backbones at an even faster inference speed. The source code is available at https://github.com/Ackesnal/GTP-ViT.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving. Builds on 38 of its 52 references that are in the map. First posted on arXiv 6 Nov 2023.

Builds on

38 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Token Merging: Your ViT But Faster17 Oct
  2. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruning27 Dec
  3. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer3 Aug

Other references · 35

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2017Graph Attention Networkscited by 8,854
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  7. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  8. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  9. 2021Scaling Vision Transformerscited by 993
  10. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  11. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  12. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
23 more
  1. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  2. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  3. 2021Twins: Revisiting the Design of Spatial Attention in Vision Transformerscited by 565
  4. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  5. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  6. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  7. 2021DeepViT: Towards Deeper Vision Transformercited by 288
  8. 2021Mobile-Former: Bridging MobileNet and Transformercited by 234
  9. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  10. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  11. 2021Adaptive Token Sampling For Efficient Vision Transformerscited by 181
  12. 2022DaViT: Dual Attention Vision Transformerscited by 174
  13. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  14. 2021Visformer: The Vision-friendly Transformercited by 137
  15. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  16. 2021RegionViT: Regional-to-Local Attention for Vision Transformerscited by 126
  17. 2021Token Pooling in Vision Transformerscited by 68
  18. 2021DPT: Deformable Patch-based Transformer for Visual Recognitioncited by 43
  19. 2022PatchDropout: Economizing Vision Transformers Using Patch Dropoutcited by 28
  20. 2021Self-slimmed Vision Transformercited by 27
  21. 2022Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Trainingcited by 24
  22. 2022Enriched CNN-Transformer Feature Aggregation Networks for Super-Resolutioncited by 22
  23. 2016Graph Summarization Methods and Applications: A Surveycited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 9; 2026: 6.

19 papers.

Sort:

By area:

  1. 2025Similarity-Aware Token Pruning: Your VLM but Fastercited by 32
  2. 2024Token Cropr: Faster ViTs for Quite a Few Taskscited by 18
  3. 2024Conditional computation in neural networks: principles and research trendscited by 10
  4. 2024PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inferencecited by 10
  5. 2024PaPr: Training-Free One-Step Patch Pruning with Lightweight ConvNets for Faster Inferencecited by 9
  6. 2025A Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusioncited by 5
  7. 2025RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layerscited by 5
  8. 2025VISA: Group-wise Visual Token Selection and Aggregation via Graph Summarization for Efficient MLLMs Inferencecited by 5
  9. 2025How Many Tokens Do 3D Point Cloud Transformer Architectures Really Need?cited by 4
  10. 2026RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inferencecited by 4
  11. 2025BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compressioncited by 2
  12. 2025CubistMerge: Spatial-Preserving Token Merging For Diverse ViT Backbonescited by 0
7 more
  1. 2025ClustViT: Clustering-based Token Merging for Semantic Segmentationcited by 0
  2. 2025Neighbor-Aware Token Reduction via Hilbert Curve for Vision Transformerscited by 0
  3. 2026RRISE: Robust Radius Inference via a Surrogate Estimatorcited by 0
  4. 2026Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspectivecited by 0
  5. 2026TORINO: Token Reduction via Interpretable Concept Overlap in Vision-Language Modelscited by 0
  6. 2026TAP-Path: Task-Adaptive Structural and Token Pruning for Efficient and Trustworthy Pathology Foundation Modelscited by 0
  7. 2026Decoder-Agnostic Token Merging for Vision Transformers: A Systematic Study of G2TMcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024TPC-ViT: Token Propagation Controller for Efficient Vision Transformer
  2. 2023No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling
  3. 2023PPT: Token Pruning and Pooling for Efficient Vision Transformers
  4. 2023Token Fusion: Bridging the Gap between Token Pruning and Token Merging
  5. 2025Token Compression Meets Compact Vision Transformers: A Survey and Comparative Evaluation for Edge AI
  6. 2026RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT
  7. 2024ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition
  8. 2023MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers