mlsanityData to 7 Oct 2026

FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance

arXiv 2501.02430 ↗First posted 5 Jan 2025cs.CVcited by 31

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Recently, Multi-modal Large Language Models (MLLMs) have shown remarkable effectiveness for multi-modal tasks due to their abilities to generate and understand cross-modal data. However, processing long sequences of visual tokens extracted from visual backbones poses a challenge for deployment in real-time applications. To address this issue, we introduce FOLDER, a simple yet effective plug-and-play module designed to reduce the length of the visual token sequence, mitigating both computational and memory demands during training and inference. Through a comprehensive analysis of the token reduction process, we analyze the information loss introduced by different reduction strategies and develop FOLDER to preserve key information while removing visual redundancy. We showcase the effectiveness of FOLDER by integrating it into the visual backbone of several MLLMs, significantly accelerating the inference phase. Furthermore, we evaluate its utility as a training accelerator or even performance booster for MLLMs. In both contexts, FOLDER achieves comparable or even better performance than the original models, while dramatically reducing complexity by removing up to 70% of visual tokens.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 92 of its 111 references that are in the map. First posted on arXiv 5 Jan 2025.

Builds on

92 of 111 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Models5 Dec
  2. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning4 Dec
  3. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification1 Dec
  4. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models30 Nov
  5. 2024Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Model16 Nov
  6. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction22 Oct
  7. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference6 Oct
  8. 2024TokenPacker: Efficient Visual Projector for Multimodal LLM2 Jul
  9. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Models18 Jun
  10. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  11. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar
  12. 2023Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Models12 Dec
2 more
  1. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers27 May
  2. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning14 Oct

Other references · 78

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
38 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  12. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  13. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  14. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  15. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  16. 2022Token Merging: Your ViT But Fastercited by 995
  17. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  18. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  19. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  20. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  21. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  22. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  23. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  24. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  25. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  26. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  27. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  28. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  29. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  30. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  31. 2021InfographicVQAcited by 430
  32. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  33. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  34. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  35. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  36. 2024What matters when building vision-language models?cited by 321
  37. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  38. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 16.

31 papers.

Sort:

By area:

  1. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  2. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Modelscited by 42
  3. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  4. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  5. 2025Efficient Multi-modal Large Language Models via Progressive Consistency Distillationcited by 22
  6. 2025TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Modelscited by 19
  7. 2026VisionTrim: Unified Vision Token Compression for Training-Free MLLM Accelerationcited by 16
  8. 2025Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Priorcited by 15
  9. 2025VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMscited by 11
  10. 2025When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMscited by 11
  11. 2026ApET: Approximation-Error Guided Token Compression for Efficient VLMscited by 11
  12. 2025StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understandingcited by 9
19 more
  1. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 8
  2. 2025METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Modelscited by 3
  3. 2025Explore More, Learn Better: Parallel MLLM Embeddings under Mutual Information Minimizationcited by 3
  4. 2026Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMscited by 3
  5. 2025Squeeze Out Tokens from Sample for Finer-Grained Data Governancecited by 2
  6. 2025Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspectivecited by 2
  7. 2026Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unificationcited by 2
  8. 2026LRCP: Low-Rank Compressibility Guided Visual Token Pruning for Efficient LVLMscited by 2
  9. 2026Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Modelscited by 2
  10. 2025Wave-Particle (Continuous-Discrete) Dualistic Visual Tokenization for Unified Understanding and Generationcited by 1
  11. 2026EntropyPrune: Matrix Entropy Guided Visual Token Pruning for Multimodal Large Language Modelscited by 1
  12. 2026VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendationcited by 0
  13. 2026POINTS-Long: Adaptive Dual-Mode Visual Reasoning in MLLMscited by 0
  14. 2026TORINO: Token Reduction via Interpretable Concept Overlap in Vision-Language Modelscited by 0
  15. 2026SA-GEM: Scale-Adaptive and Geospatial Evidence-Modulated Token Pruning for Efficient Remote Sensing Large Vision-Language Modelscited by 0
  16. 2026Position Is All You Need: A Free Lunch Token Compression Strategy for MLLM-based Referring Expression Segmentationcited by 0
  17. 2026Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generationcited by 0
  18. 2026ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compressioncited by 0
  19. 2026DIPrune: Task-Aware Token Pruning with Dual Importance for Efficient Multimodal Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
  2. 2026Fast-Slow Efficient Training for Multimodal Large Language Models via Visual Token Pruning
  3. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming
  4. 2024The Revolution of Multimodal Large Language Models: A Survey
  5. 2026Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelines
  6. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference
  7. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy
  8. 2025LFTR: Learning-Free Token Reduction for Multimodal Large Language Models