mlsanityData to 7 Oct 2026

MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer

arXiv 2403.02991 ↗First posted 5 Mar 2024cs.CVcited by 53

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

Vision-Language Transformers (VLTs) have shown great success recently, but are meanwhile accompanied by heavy computation costs, where a major reason can be attributed to the large number of visual and language tokens. Existing token pruning research for compressing VLTs mainly follows a single-modality-based scheme yet ignores the critical role of aligning different modalities for guiding the token pruning process, causing the important tokens for one modality to be falsely pruned in another modality branch. Meanwhile, existing VLT pruning works also lack the flexibility to dynamically compress each layer based on different input samples. To this end, we propose a novel framework named Multimodal Alignment-Guided Dynamic Token Pruning (MADTP) for accelerating various VLTs. Specifically, we first introduce a well-designed Multi-modality Alignment Guidance (MAG) module that can align features of the same semantic concept from different modalities, to ensure the pruned tokens are less important for all modalities. We further design a novel Dynamic Token Pruning (DTP) module, which can adaptively adjust the token compression ratio in each layer based on different input instances. Extensive experiments on various benchmarks demonstrate that MADTP significantly reduces the computational complexity of kinds of multimodal models while preserving competitive performance. Notably, when applied to the BLIP model in the NLVR2 dataset, MADTP can reduce the GFLOPs by 80% with less than 4% performance degradation.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 44 of its 50 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

44 of 50 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023ELIP: Efficient Discriminative Language-Image Pre-training with Fewer Vision Tokens28 Sep

Other references · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
31 more
  1. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  2. 2020Knowledge Distillation: A Surveycited by 1,578
  3. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  5. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  6. 2022Token Merging: Your ViT But Fastercited by 995
  7. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  8. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  9. 2020What is the State of Neural Network Pruning?cited by 578
  10. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  11. 2021Dynamic Neural Networks: A Surveycited by 405
  12. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  13. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  14. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  15. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  16. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  17. 2021AdaViT: Adaptive Vision Transformers for Efficient Image Recognitioncited by 228
  18. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  19. 2015Structured Pruning of Deep Convolutional Neural Networkscited by 148
  20. 2021Patch Slimming for Efficient Vision Transformerscited by 131
  21. 2022PMR: Prototypical Modal Rebalance for Multimodal Learningcited by 114
  22. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  23. 2021Global Vision Transformer Pruning with Hessian-Aware Saliencycited by 69
  24. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  25. 2023X-Pruner: eXplainable Pruning for Vision Transformerscited by 66
  26. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 51
  27. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  28. 2022You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Modelcited by 42
  29. 2022Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attentioncited by 35
  30. 2020Fully Dynamic Inference with Deep Neural Networkscited by 19
  31. 2023Multi-Scale And Token Mergence: Make Your ViT More Efficientcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 28; 2026: 17.

53 papers.

Sort:

By area:

  1. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters Morecited by 149
  2. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  3. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  4. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Modelscited by 89
  5. 2025VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Cachingcited by 83
  6. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  7. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  8. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  9. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
  10. 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMscited by 23
  11. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  12. 2024FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compressioncited by 21
38 more
  1. 2025TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Modelscited by 19
  2. 2025Learning to Inference Adaptively for Multimodal Large Language Modelscited by 12
  3. 2024Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Drivingcited by 11
  4. 2025Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skippingcited by 11
  5. 2025GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awarenesscited by 11
  6. 2026The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gatingcited by 11
  7. 2026Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Modelscited by 9
  8. 2025MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understandingcited by 7
  9. 2025GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMscited by 6
  10. 2025ViCO: A Training Strategy towards Semantic Aware Dynamic High-Resolutioncited by 6
  11. 2025FlowMM: Cross-Modal Information Flow Guided KV Cache Merging for Efficient Multimodal Context Inferencecited by 6
  12. 2025Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generationcited by 5
  13. 2024PAR: Prompt-Aware Token Reduction Method for Efficient Large Multimodal Modelscited by 4
  14. 2025Hierarchical Safety Realignment: Lightweight Restoration of Safety in Pruned Large Vision-Language Modelscited by 4
  15. 2025CLIP-MG: Guiding Semantic Attention with Skeletal Pose Features and RGB Data for Micro-Gesture Recognition on the iMiGUE Datasetcited by 4
  16. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Modelscited by 4
  17. 2025ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Modelscited by 4
  18. 2025MCE: Towards a General Framework for Handling Missing Modalities under Imbalanced Missing Ratescited by 4
  19. 2026Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Modelscited by 4
  20. 2025Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMscited by 3
  21. 2025DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understandingcited by 2
  22. 2025STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inferencecited by 2
  23. 2026Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understandingcited by 2
  24. 2026MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMscited by 2
  25. 2025LFTR: Learning-Free Token Reduction for Multimodal Large Language Modelscited by 1
  26. 2025Dynamic Token Reweighting for Robust Vision-Language Modelscited by 1
  27. 2025Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understandingcited by 1
  28. 2026Learning to Accelerate Vision-Language-Action Models through Adaptive Visual Token Cachingcited by 1
  29. 2026An Efficient Token Compression Framework for Visual Object Trackingcited by 1
  30. 2025In-Context Clustering with Large Language Modelscited by 0
  31. 2025OS-HGAdapter: Open Semantic Hypergraph Adapter for Large Language Models Assisted Entropy-Enhanced Image-Text Alignmentcited by 0
  32. 2026CLIP-Map: Structured Matrix Mapping for Parameter-Efficient CLIP Compressioncited by 0
  33. 2026ETA-VLA: Efficient Token Adaptation via Temporal Fusion and Intra-LLM Sparsification for Vision-Language-Action Modelscited by 0
  34. 2026Efficient3D: A Unified Framework for Adaptive and Debiased Token Reduction in 3D MLLMscited by 0
  35. 2026VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Modelscited by 0
  36. 2026Inconsistency-aware Multimodal Schr"odinger Bridge for Deepfake Localizationcited by 0
  37. 2026Uncertainty-DTW for Sequences and Visual Tokenscited by 0
  38. 2026Development of Vision-Language Model-based GNSS Spoofing Detection for Autonomous Vehicle Navigationcited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Similarity-Aware Token Pruning: Your VLM but Faster
  2. 2025Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization
  3. 2025TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models
  4. 2026Multi-Image Visual Token Pruning in Large Visual Language Models
  5. 2025LVPruning: An Effective yet Simple Language-Guided Vision Token Pruning Approach for Multi-modal Large Language Models
  6. 2024VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation
  7. 2026Collaborative Multi-Mode Pruning for Vision-Language Models
  8. 2026ERASE: Eliminating Redundant Visual Tokens via Adaptive Two-Stage Token Pruning