mlsanityData to 14 Oct 2026

VoCo-LLaMA: Towards Vision Compression with Large Language Models

arXiv 2406.12275 ↗First posted 18 Jun 2024cs.CVcited by 79

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Vision-Language Models (VLMs) have achieved remarkable success in various multi-modal tasks, but they are often bottlenecked by the limited context window and high computational cost of processing high-resolution image inputs and videos. Vision compression can alleviate this problem by reducing the vision token count. Previous approaches compress vision tokens with external modules and force LLMs to understand the compressed ones, leading to visual information loss. However, the LLMs' understanding paradigm of vision tokens is not fully utilised in the compression learning process. We propose VoCo-LLaMA, the first approach to compress vision tokens using LLMs. By introducing Vision Compression tokens during the vision instruction tuning phase and leveraging attention distillation, our method distill how LLMs comprehend vision tokens into their processing of VoCo tokens. VoCo-LLaMA facilitates effective vision compression and improves the computational efficiency during the inference stage. Specifically, our method achieves minimal performance loss with a compression ratio of 576×, resulting in up to 94.8% fewer FLOPs and 69.6% acceleration in inference time. Furthermore, through continuous training using time-series compressed token sequences of video frames, VoCo-LLaMA demonstrates the ability to understand temporal correlations, outperforming previous methods on popular video question-answering benchmarks. Our approach presents a promising way to unlock the full potential of VLMs' contextual window, enabling more scalable multi-modal applications. The project page, along with the associated code, can be accessed via https://yxxxb.github.io/VoCo-LLaMA-page/.

Key facts. Cited by 79 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 69 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

50 of 69 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric12 Mar
  2. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  12. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
36 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  11. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  13. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  14. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  15. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  16. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  17. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  18. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  19. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  20. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  21. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  22. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  23. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  24. 2023Adapting Language Models to Compress Contextscited by 356
  25. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  26. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  27. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 255
  28. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  29. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  30. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  31. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  32. 2022Learning by Distilling Contextcited by 138
  33. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 97
  34. 2021Poolingformer: Long Document Modeling with Pooling Attentioncited by 60
  35. 2023BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuningcited by 41
  36. 2023Segment and Caption Anythingcited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 38; 2026: 27.

79 papers.

Sort:

By area:

  1. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 391
  2. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 213
  3. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 176
  4. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 166
  5. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 84
  6. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 81
  7. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 68
  8. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
  9. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  10. 2025KV-Edit: Training-Free Image Editing for Precise Background Preservationcited by 49
  11. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 49
  12. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
38 more
  1. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  2. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentationcited by 34
  3. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Modelscited by 32
  4. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 30
  5. 2024Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reductioncited by 28
  6. 2024[CLS] Token Tells Everything Needed for Training-free Efficient MLLMscited by 26
  7. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 26
  8. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  9. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 23
  10. 2025Efficient Multi-modal Large Language Models via Progressive Consistency Distillationcited by 22
  11. 2025GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignmentcited by 21
  12. 2025FrameMind: Frame-Interleaved Video Reasoning via Reinforcement Learningcited by 21
  13. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Controlcited by 20
  14. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimmingcited by 20
  15. 2025Auto-scaling Continuous Memory for GUI Agentcited by 20
  16. 2025Hybrid Token Compression for Vision-Language Modelscited by 20
  17. 2026VisionTrim: Unified Vision Token Compression for Training-Free MLLM Accelerationcited by 18
  18. 2026HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exitcited by 18
  19. 2025Towards General Continuous Memory for Vision-Language Modelscited by 15
  20. 2025ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layerscited by 14
  21. 2026COMI: Coarse-to-fine Context Compression via Marginal Information Gaincited by 13
  22. 2025Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skippingcited by 12
  23. 2026IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruningcited by 12
  24. 2024VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentationcited by 11
  25. 2025DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMscited by 10
  26. 2024Hierarchical Memory for Long Video QAcited by 9
  27. 2024Efficient Multi-modal Large Language Models via Visual Token Groupingcited by 9
  28. 2025Widget2Code: From Visual Widgets to UI Code via Multimodal LLMscited by 9
  29. 2025CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Modelscited by 8
  30. 2025GEM: Empowering LLM for both Embedding Generation and Language Understandingcited by 8
  31. 2024FM2DS: Few-Shot Multimodal Multihop Data Synthesis with Knowledge Distillation for Question Answeringcited by 7
  32. 2025EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Modelscited by 7
  33. 2025Beyond Intermediate States: Explaining Visual Redundancy through Languagecited by 7
  34. 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoningcited by 7
  35. 2026Multimodal Model for Computational Pathology:Representation Learning and Image Compressioncited by 6
  36. 2025LEO-MINI: An Efficient Multimodal Large Language Model using Conditional Token Reduction and Mixture of Multi-Modal Expertscited by 4
  37. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Modelscited by 4
  38. 2025TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understandingcited by 4

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression
  2. 2025Variation-aware Vision Token Dropping for Faster Large Vision-Language Models
  3. 2025LLMC+: Benchmarking Vision-Language Model Compression with a Plug-and-play Toolkit
  4. 2026ApET: Approximation-Error Guided Token Compression for Efficient VLMs
  5. 2026One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding
  6. 2026Vision Token Reduction via Attention-Driven Self-Compression for Efficient Multimodal Large Language Models
  7. 2025Towards Lossless Ultimate Vision Token Compression for VLMs
  8. 2024Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters