mlsanityData to 7 Oct 2026

On the Limits of Token Reduction for Efficient Unified Vision Language Training

arXiv 2606.01503 ↗First posted 31 May 2026cs.CV cs.AI cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Unified vision-language models (VLMs) integrate visual understanding and visual generation within a single autoregressive backbone, but their joint training is computationally expensive and largely overlooked from an efficiency perspective. In this work, we study the feasibility and limits of token-reduction-based acceleration for unified VLM training. Through a systematic analysis of layerwise attention allocation, we uncover a fundamental asymmetry: visual understanding exhibits substantial late-layer visual redundancy, whereas visual generation maintains persistent dependence on image tokens across depth. Guided by this observation, we design task-specific accelerators that selectively reduce image-token computation for each objective. While these methods achieve significant efficiency gains in isolated settings, we observe a consistent synergy loss under unified training -- task-specific token dropping necessitates divergent parameter pathways and eliminates the mutual performance gains typically observed in joint optimization. Our findings suggest that efficient unified modeling requires preserving shared cross-task structures, highlighting the need for synergy-aware acceleration strategies. Project page: https://chicychen.github.io/TokenReductionUnifiedVLM/.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 38 references that are in the map. First posted on arXiv 31 May 2026.

Builds on

32 of 38 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token7 Jan
  2. 2024A-VL: Adaptive Attention for Large Vision-Language Models23 Sep
  3. 2024TokenPacker: Efficient Visual Projector for Multimodal LLM2 Jul
  4. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference9 May
  5. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  6. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  11. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
14 more
  1. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  2. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  3. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  4. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  5. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  6. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  7. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 249
  8. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  9. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  10. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  11. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  12. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  13. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenizationcited by 87
  14. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 62

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Accelerating Unified Multimodal Models with Core-Expansion Routing and Unified Computation Schedulingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference
  2. 2026Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models
  3. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models
  4. 2025TokenFLEX: Unified VLM Training for Flexible Visual Tokens Inference
  5. 2026Do Vision Language Models Need to Process Image Tokens?
  6. 2025Cross-Task Generalization Between Understanding and Generation in Unified Vision-Language Models: A Controlled Study
  7. 2025HERO: Rethinking Visual Token Early Dropping in High-Resolution Large Vision-Language Models
  8. 2026LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models