mlsanityData to 7 Oct 2026

CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers

arXiv 2305.17455 ↗First posted 27 May 2023cs.CV cs.CLcited by 44

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Recent vision-language models have achieved tremendous advances. However, their computational costs are also escalating dramatically, making model acceleration exceedingly critical. To pursue more efficient vision-language Transformers, this paper introduces Cross-Guided Ensemble of Tokens (CrossGET), a general acceleration framework for vision-language Transformers. This framework adaptively combines tokens in real-time during inference, significantly reducing computational costs while maintaining high performance. CrossGET features two primary innovations: 1) Cross-Guided Matching and Ensemble. CrossGET leverages cross-modal guided token matching and ensemble to effectively utilize cross-modal information, achieving wider applicability across both modality-independent models, e.g., CLIP, and modality-dependent ones, e.g., BLIP2. 2) Complete-Graph Soft Matching. CrossGET introduces an algorithm for the token-matching mechanism, ensuring reliable matching results while facilitating parallelizability and high efficiency. Extensive experiments have been conducted on various vision-language tasks, such as image-text retrieval, visual reasoning, image captioning, and visual question answering. The performance on both classic multimodal architectures and emerging multimodal LLMs demonstrates the framework's effectiveness and versatility. The code is available at https://github.com/sdc17/CrossGET.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 92 of its 123 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

92 of 123 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning14 Oct

Other references · 91

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  12. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  13. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  14. 2013Describing Textures in the Wildcited by 2,110
  15. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  16. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  17. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  18. 2019Towards VQA Models That Can Readcited by 1,696
  19. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  20. 2022Visual Prompt Tuningcited by 1,674
  21. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  22. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  23. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  24. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  25. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  26. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  27. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  28. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  29. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  30. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  31. 2017Mixed Precision Trainingcited by 1,135
  32. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  33. 2022Token Merging: Your ViT But Fastercited by 995
  34. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  35. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  36. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  37. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  38. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 14; 2025: 16; 2026: 11.

44 papers.

Sort:

By area:

  1. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  2. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  3. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  4. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 153
  5. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 137
  6. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 53
  7. 2023TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understandingcited by 49
  8. 2024Accelerating Transformers with Spectrum-Preserving Token Mergingcited by 35
  9. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 31
  10. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  11. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 30
  12. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
32 more
  1. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25
  2. 2025LVPruning: An Effective yet Simple Language-Guided Vision Token Pruning Approach for Multi-modal Large Language Modelscited by 21
  3. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimmingcited by 20
  4. 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 20
  5. 2024Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Informationcited by 18
  6. 2025Superficial Self-Improved Reasoners Benefit from Model Mergingcited by 16
  7. 2026VisionTrim: Unified Vision Token Compression for Training-Free MLLM Accelerationcited by 16
  8. 2023Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Modelscited by 11
  9. 2026Behavior Knowledge Merge in Reinforced Agentic Modelscited by 10
  10. 2026LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Modelscited by 10
  11. 2024AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularitycited by 9
  12. 2025LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Modelscited by 9
  13. 2024freePruner: A Training-free Approach for Large Multimodal Model Accelerationcited by 7
  14. 2024Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuningcited by 5
  15. 2025QG-VTC: Question-Guided Visual Token Compression in MLLMs for Efficient VQAcited by 4
  16. 2025CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Modelscited by 4
  17. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Modelscited by 4
  18. 2025ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Modelscited by 4
  19. 2025Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Modelscited by 3
  20. 2026StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Modelscited by 2
  21. 2026Accelerating Multimodal Large Language Models with Prior-Corrected Token Reductioncited by 2
  22. 2024NAVERO: Unlocking Fine-Grained Semantics for Video-Language Compositionalitycited by 1
  23. 2024Sparsity Meets Similarity: Leveraging Long-Tail Distribution for Dynamic Optimized Token Representation in Multimodal Large Language Modelscited by 1
  24. 2025LFTR: Learning-Free Token Reduction for Multimodal Large Language Modelscited by 1
  25. 2025Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understandingcited by 1
  26. 2026Spectral Evolution-Guided Token Pruning in Multimodal Large Language Modelscited by 1
  27. 2025Dynamic Pyramid Network for Efficient Multimodal Large Language Modelcited by 0
  28. 2026Q Cache: Visual Attention is Valuable in Less than Half of Decode Layers for Multimodal Large Language Modelcited by 0
  29. 2026CLIP-Map: Structured Matrix Mapping for Parameter-Efficient CLIP Compressioncited by 0
  30. 2026Efficient3D: A Unified Framework for Adaptive and Debiased Token Reduction in 3D MLLMscited by 0
  31. 2026BMFA: Boundary-Minority Free-Energy Adaptive Screeningcited by 0
  32. 2026MAViE: A Multi-scale Adaptive Vision Encoder for Fine-grained Visual Perception and Efficient Multimodal Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval
  2. 2024Cross-Modal Consistency in Multimodal Large Language Models
  3. 2026CoRe: A Comprehensive Framework for Cross-Image Comparative Reasoning in Vision-Language Models
  4. 2025CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion
  5. 2025MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching
  6. 2025Token Sequence Compression for Efficient Multimodal Computing
  7. 2024Towards Cross-modal Backward-compatible Representation Learning for Vision-Language Models
  8. 2022COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval