mlsanityData to 7 Oct 2026

TokenPacker: Efficient Visual Projector for Multimodal LLM

arXiv 2407.02392 ↗First posted 2 Jul 2024cs.CVcited by 170

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

The visual projector serves as an essential bridge between the visual encoder and the Large Language Model (LLM) in a Multimodal LLM (MLLM). Typically, MLLMs adopt a simple MLP to preserve all visual contexts via one-to-one transformation. However, the visual tokens are redundant and can be considerably increased when dealing with high-resolution images, impairing the efficiency of MLLMs significantly. Some recent works have introduced resampler or abstractor to reduce the number of resulting visual tokens. Unfortunately, they fail to capture finer details and undermine the visual reasoning capabilities of MLLMs. In this work, we propose a novel visual projector, which adopts a coarse-to-fine scheme to inject the enriched characteristics to generate the condensed visual tokens. In specific, we first interpolate the visual features as a low-resolution point query, providing the overall visual representation as the foundation. Then, we introduce a region-to-point injection module that utilizes high-resolution, multi-level region-based cues as fine-grained reference keys and values, allowing them to be fully absorbed within the corresponding local context region. This step effectively updates the coarse point query, transforming it into an enriched one for the subsequent LLM reasoning. Extensive experiments demonstrate that our approach compresses the visual tokens by 75%~89%, while achieves comparable or even better performance across diverse benchmarks with significantly higher efficiency. The source codes can be found at https://github.com/CircleRadon/TokenPacker.

Key facts. Cited by 170 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 72 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

56 of 72 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Matryoshka Query Transformer for Large Vision-Language Models29 May
  2. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  3. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images18 Mar

Other references · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Segment Anythingcited by 8,158
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022A ConvNet for the 2020scited by 3,545
38 more
  1. 2023Qwen Technical Reportcited by 3,273
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  13. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  14. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  15. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  16. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  17. 2022Token Merging: Your ViT But Fastercited by 995
  18. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  19. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  20. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  21. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  22. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  23. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  24. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  25. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  26. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  27. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  28. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  29. 2022Visual Spatial Reasoningcited by 288
  30. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  31. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  32. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  33. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  34. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  35. 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 158
  36. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  37. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Modelcited by 143
  38. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 96

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 90; 2026: 54.

170 papers.

Sort:

By area:

  1. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  2. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
  3. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  4. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  5. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 113
  6. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  7. 2025Efficient Reasoning Models: A Surveycited by 93
  8. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  9. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  10. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 71
  11. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 70
  12. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
38 more
  1. 2024SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizercited by 55
  2. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  3. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  4. 2024Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Accelerationcited by 38
  5. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 37
  6. 2025POSTA: A Go-to Framework for Customized Artistic Poster Generationcited by 36
  7. 2025When Large Vision-Language Model Meets Large Remote Sensing Imagery: Coarse-to-Fine Text-Guided Token Pruningcited by 35
  8. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Searchcited by 34
  9. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  10. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 31
  11. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  12. 2024Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reductioncited by 28
  13. 2025PixelThink: Towards Efficient Chain-of-Pixel Reasoningcited by 28
  14. 2024DocVLM: Make Your VLM an Efficient Readercited by 27
  15. 2024Inference Optimal VLMs Need Fewer Visual Tokens and More Parameterscited by 25
  16. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25
  17. 2025Variation-aware Vision Token Dropping for Faster Large Vision-Language Modelscited by 25
  18. 2025Efficient Multi-modal Large Language Models via Progressive Consistency Distillationcited by 22
  19. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to Seecited by 21
  20. 2025InternVL-X: Advancing and Accelerating InternVL Series with Efficient Visual Token Compressioncited by 21
  21. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21
  22. 2024Importance-Based Token Merging for Efficient Image and Video Generationcited by 20
  23. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimmingcited by 20
  24. 2025EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?cited by 20
  25. 2025FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruningcited by 20
  26. 2024Order Matters: Exploring Order Sensitivity in Multimodal Large Language Modelscited by 19
  27. 2025Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Modelscited by 19
  28. 2025Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inferencecited by 18
  29. 2026VisionTrim: Unified Vision Token Compression for Training-Free MLLM Accelerationcited by 16
  30. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findingscited by 15
  31. 2025Collaboration of Large Language Models and Small Recommendation Models for Device-Cloud Recommendationcited by 15
  32. 2025Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Drivingcited by 15
  33. 2026HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exitcited by 15
  34. 2024VMAD: Visual-enhanced Multimodal Large Language Model for Zero-Shot Anomaly Detectioncited by 13
  35. 2025A Token-level Text Image Foundation Model for Document Understandingcited by 13
  36. 2025ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layerscited by 13
  37. 2026N"uwa: Mending the Spatial Integrity Torn by VLM Token Pruningcited by 13
  38. 2025Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrievalcited by 12

50 most cited of 170 All 170 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Slot-MLLM: Object-Centric Visual Tokenization for Multimodal LLM
  2. 2024DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models
  3. 2025LLaVA-SP: Enhancing Visual Representation with Visual Spatial Tokens for MLLMs
  4. 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
  5. 2024From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
  6. 2024Instruction Tuning-free Visual Token Complement for Multimodal LLMs
  7. 2025Token Sequence Compression for Efficient Multimodal Computing
  8. 2025Window Token Concatenation for Efficient Visual Large Language Models