mlsanityData to 7 Oct 2026

DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs

arXiv 2406.04334 ↗First posted 6 Jun 2024cs.CVcited by 72

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Most large multimodal models (LMMs) are implemented by feeding visual tokens as a sequence into the first layer of a large language model (LLM). The resulting architecture is simple but significantly increases computation and memory costs, as it has to handle a large number of additional tokens in its input layer. This paper presents a new architecture DeepStack for LMMs. Considering N layers in the language and vision transformer of LMMs, we stack the visual tokens into N groups and feed each group to its aligned transformer layer textitfrom bottom to top. Surprisingly, this simple method greatly enhances the power of LMMs to model interactions among visual tokens across layers but with minimal additional cost. We apply DeepStack to both language and vision transformer in LMMs, and validate the effectiveness of DeepStack LMMs with extensive empirical results. Using the same context length, our DeepStack 7B and 13B parameters surpass their counterparts by 2.7 and 2.9 on average across 9 benchmarks, respectively. Using only one-fifth of the context length, DeepStack rivals closely to the counterparts that use the full context length. These gains are particularly pronounced on high-resolution tasks, e.g., 4.2, 11.0, and 4.0 improvements on TextVQA, DocVQA, and InfoVQA compared to LLaVA-1.5-7B, respectively. We further apply DeepStack to vision transformer layers, which brings us a similar amount of improvements, 3.8 on average compared with LLaVA-1.5-7B.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 62 of its 81 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

62 of 81 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
38 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  13. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  14. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  15. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  16. 2019Towards VQA Models That Can Readcited by 1,696
  17. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  18. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  19. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  20. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  21. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  22. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  23. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  24. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  25. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  26. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  27. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  28. 2016A Diagram Is Worth A Dozen Imagescited by 793
  29. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  30. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  31. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  32. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  33. 2023Instruction Tuning with GPT-4cited by 676
  34. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  35. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  36. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  37. 2023Textbooks Are All You Needcited by 561
  38. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 16; 2026: 47.

72 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  3. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  4. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  5. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  6. 2026STEP3-VL-10B Technical Reportcited by 28
  7. 2024[CLS] Token Tells Everything Needed for Training-free Efficient MLLMscited by 26
  8. 2025Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practicescited by 21
  9. 2025TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Modelscited by 19
  10. 2025UniVid: The Open-Source Unified Video Modelcited by 18
  11. 2024REDUCIO! Generating 1K Video within 16 Seconds using Extremely Compressed Motion Latentscited by 14
  12. 2026FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Mergingcited by 14
38 more
  1. 2024VMAD: Visual-enhanced Multimodal Large Language Model for Zero-Shot Anomaly Detectioncited by 13
  2. 2026Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation Systemcited by 10
  3. 2024TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Modelscited by 9
  4. 2026FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selectioncited by 9
  5. 2024INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuningcited by 8
  6. 2025Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Modelscited by 8
  7. 2026From Pixels to Words -- Towards Native One-Vision Models at Scalecited by 6
  8. 2026Visual Jev: Accurate and Efficient Decisions from Shared Visual Contextcited by 6
  9. 2025Enhancing Vision Foundation Models via Multimodal Continual Pre-Trainingcited by 5
  10. 2026VideoLoom: A Video Large Language Model for Joint Spatial-Temporal Understandingcited by 5
  11. 2024INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal Large Language Modelcited by 4
  12. 2025The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Losscited by 4
  13. 2026Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Modelscited by 4
  14. 2026U-VLM: Hierarchical Vision Language Modeling for Report Generationcited by 4
  15. 2025METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Modelscited by 3
  16. 2026Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMscited by 3
  17. 2026Personalize Your Large Vision-language Models With In-context Prompt Tuningcited by 3
  18. 2026Earth-OneVision: Extending Remote Sensing Multimodal Large Language Models to More Sensor Modalities and Taskscited by 3
  19. 2026Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unificationcited by 3
  20. 2025Vision Remember: Recovering Visual Information in Efficient LVLM with Vision Feature Resamplingcited by 2
  21. 2026CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Modelscited by 2
  22. 2026Don't Blink: Evidence Collapse during Multimodal Reasoningcited by 2
  23. 2026SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inferencecited by 2
  24. 2026Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Modelscited by 2
  25. 2025Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understandingcited by 1
  26. 2025Mitigating Information Loss under High Pruning Rates for Efficient Large Vision Language Modelscited by 1
  27. 2025VAT: Vision Action Transformer by Unlocking Full Representation of ViTcited by 1
  28. 2026From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusioncited by 1
  29. 2026Where and How to Prune: An Empirical Study of Visual Token Pruning for GUI Agent Navigationcited by 1
  30. 2026Sparse Autoencoders as Plug-and-Play Firewalls for Adversarial Attack Detection in VLMscited by 1
  31. 2026Instruction Lens Score: Your Instruction Contributes a Powerful Object Hallucination Detector for Multimodal Large Language Modelscited by 1
  32. 2026Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomographycited by 1
  33. 2024A Study on Context Length and Efficient Transformers for Biomedical Image Analysiscited by 0
  34. 2025Towards Open World Detection: A Surveycited by 0
  35. 2025InstructPLM-mu: 1-Hour Fine-Tuning of ESM2 Beats ESM3 in Protein Mutation Predictionscited by 0
  36. 2026MathDoc: Benchmarking Structured Extraction and Active Refusal on Noisy Mathematics Exam Paperscited by 0
  37. 2026Sparse Shortcuts: Facilitating Efficient Fusion in Multimodal Large Language Modelscited by 0
  38. 2026Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Modelscited by 0

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token
  2. 2024iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models
  3. 2025LVPruning: An Effective yet Simple Language-Guided Vision Token Pruning Approach for Multi-modal Large Language Models
  4. 2025TrimTokenator: Towards Adaptive Visual Token Pruning for Large Multimodal Models
  5. 2025TrimTokenator-LC: Towards Adaptive Visual Token Pruning for Large Multimodal Models with Long Contexts
  6. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models
  7. 2025Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Models
  8. 2026How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A