mlsanityData to 14 Oct 2026

ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs

arXiv 2507.21420 ↗First posted 29 Jul 2025cs.CV cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

The computational cost of training multimodal large language models (MLLMs) grows rapidly with the number of processed tokens. Existing efficiency methods mainly target inference via token reduction or merging, offering limited benefits during training. We introduce ReGATE (Reference-Guided Adaptive Token Elision), an adaptive token pruning method for accelerating MLLM training. ReGATE adopts a teacher-student framework, in which a frozen teacher LLM provides per-token guidance losses that are fused with an exponential moving average of the student's difficulty estimates. This adaptive scoring mechanism dynamically selects informative tokens while skipping redundant ones in the forward pass, substantially reducing computation without altering the model architecture. Across three representative MLLMs, ReGATE matches the peak accuracy of standard training on MVBench up to 2× faster, using only 38% of the tokens. With extended training, it even surpasses the baseline across multiple multimodal benchmarks, cutting total token usage by over 41%.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 68 references that are in the map. First posted on arXiv 29 Jul 2025.

Builds on

55 of 68 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025LaVi: Efficient Large Vision-Language Models via Internal Feature Modulation20 Jun
  2. 2025TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models14 Apr
  3. 2025PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models11 Apr
  4. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model24 Mar
  5. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models4 Mar
  6. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models30 Nov
  7. 2024HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models20 Aug
  8. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  10. 2024Qwen2 Technical Reportcited by 2,120
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
35 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  4. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  5. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  6. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  7. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  8. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  9. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  10. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  11. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  12. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  13. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  14. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  15. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  16. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
  17. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  18. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
  19. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  20. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  21. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 96
  22. 2023Dynamic Token Pruning in Plain Vision Transformers for Semantic Segmentationcited by 62
  23. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 40
  24. 2024Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videoscited by 38
  25. 2025Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMscited by 31
  26. 2023DIME-FM: DIstilling Multimodal and Efficient Foundation Modelscited by 28
  27. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillationcited by 26
  28. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 22
  29. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  30. 2023The Role of Masking for Efficient Supervised Knowledge Distillation of Vision Transformerscited by 8
  31. 2025VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignmentcited by 8
  32. 2024VideoSAVi: Self-Aligned Video Language Models without Human Supervisioncited by 7
  33. 2023Unlock the Power: Competitive Distillation for Multi-Modal Large Language Modelscited by 3
  34. 2025Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuningcited by 3
  35. 2025Static or Dynamic: Towards Query-Adaptive Token Selection for Video Question Answeringcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Improving Visual Reasoning with Iterative Evidence Refinementcited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2025From Dense to Dynamic: Token-Difficulty Driven MoEfication of Pre-Trained LLMs
  2. 2025Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?
  3. 2026Stepwise Token Selection for Efficient Multimodal Large Language Models
  4. 2026Fast-Slow Efficient Training for Multimodal Large Language Models via Visual Token Pruning
  5. 2025VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs
  6. 2026Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models
  7. 2025Improving MLLM Training Efficiency via Stage-Aware Sparsity
  8. 2026TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference