mlsanityData to 14 Oct 2026

Matryoshka Multimodal Models

arXiv 2405.17430 ↗First posted 27 May 2024cs.CV cs.AI cs.CLcited by 84

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Large Multimodal Models (LMMs) such as LLaVA have shown strong performance in visual-linguistic reasoning. These models first embed images into a fixed large number of visual tokens and then feed them into a Large Language Model (LLM). However, this design causes an excessive number of tokens for dense visual scenarios such as high-resolution images and videos, leading to great inefficiency. While token pruning/merging methods do exist, they produce a single length output for each image and do not afford flexibility in trading off information density v.s. efficiency. Inspired by the concept of Matryoshka Dolls, we propose M3: Matryoshka Multimodal Models, which learns to represent visual content as nested sets of visual tokens that capture information across multiple coarse-to-fine granularities. Our approach offers several unique benefits for LMMs: (1) One can explicitly control the visual granularity per test instance during inference, e.g. , adjusting the number of tokens used to represent an image based on the anticipated complexity or simplicity of the content; (2) M3 provides a framework for analyzing the granularity needed for existing datasets, where we find that COCO-style benchmarks only need around ~9 visual tokens to obtain accuracy similar to that of using all 576 tokens; (3) Our approach provides a foundation to explore the best trade-off between performance and visual token length at sample level, where our investigation reveals that a large gap exists between the oracle upper bound and current fixed-scale representations.

Key facts. Cited by 84 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 66 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

52 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  2. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2020Reformer: The Efficient Transformercited by 1,668
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  13. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  14. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  15. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  16. 2016Modeling Context in Referring Expressionscited by 1,063
  17. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  18. 2022Token Merging: Your ViT But Fastercited by 1,021
  19. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  20. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  21. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  22. 2016A Diagram Is Worth A Dozen Imagescited by 804
  23. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  24. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  25. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  26. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  27. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  28. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  29. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  30. 2022Matryoshka Representation Learningcited by 282
  31. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197
  32. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  33. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 161
  34. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  35. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  36. 2014Learning Ordered Representations with Nested Dropoutcited by 84
  37. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformerscited by 60
  38. 20242D Matryoshka Sentence Embeddingscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 16; 2025: 40; 2026: 28.

84 papers.

Sort:

By area:

  1. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 391
  2. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 181
  3. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 157
  4. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  5. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
  6. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 97
  7. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 96
  8. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Modelscited by 89
  9. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 77
  10. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  11. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
  12. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
38 more
  1. 2025Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videoscited by 44
  2. 2024Adaptive Length Image Tokenization via Recurrent Allocationcited by 43
  3. 2024Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Accelerationcited by 41
  4. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 39
  5. 2024DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimmingcited by 35
  6. 2025Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representationcited by 26
  7. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  8. 2025CAT: Content-Adaptive Image Tokenizationcited by 21
  9. 2024PUMA: Empowering Unified MLLM with Multi-granular Visual Generationcited by 20
  10. 2026FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Mergingcited by 18
  11. 2026N"uwa: Mending the Spatial Integrity Torn by VLM Token Pruningcited by 15
  12. 2026OAT: Ordered Action Tokenizationcited by 15
  13. 2025LongViTU: Instruction Tuning for Long-Form Video Understandingcited by 13
  14. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  15. 2025Efficient LLaMA-3.2-Vision by Trimming Cross-attended Visual Featurescited by 10
  16. 2024AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularitycited by 9
  17. 2024TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Modelscited by 9
  18. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
  19. 2025MGVQ: Could VQ-VAE Beat VAE? A Generalizable Tokenizer with Multi-group Quantizationcited by 9
  20. 2025ExAct: A Video-Language Benchmark for Expert Action Analysiscited by 8
  21. 2025CropVLM: Learning to Zoom for Fine-Grained Vision-Language Perceptioncited by 8
  22. 2024MatMamba: A Matryoshka State Space Modelcited by 7
  23. 2025ToDRE: Effective Visual Token Pruning via Token Diversity and Task Relevancecited by 7
  24. 2025Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilizationcited by 7
  25. 2025Delta-LLaVA: Base-then-Specialize Alignment for Token-Efficient Vision-Language Modelscited by 7
  26. 2024Exploring the Design Space of Visual Context Representation in Video MLLMscited by 6
  27. 2025Make Your Training Flexible: Towards Deployment-Efficient Video Modelscited by 6
  28. 2025FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understandingcited by 6
  29. 2025Exploring The Visual Feature Space for Multimodal Neural Decodingcited by 4
  30. 2025Attention Debiasing for Token Pruning in Vision Language Modelscited by 4
  31. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Modelscited by 4
  32. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  33. 2025Elastic ViTs from Pretrained Models without Retrainingcited by 4
  34. 2025Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Modelscited by 4
  35. 2026Energy-Driven Adaptive Visual Token Pruning for Efficient Vision-Language Modelscited by 4
  36. 2026EvoGround: Self-Evolving Video Agents for Video Temporal Groundingcited by 4
  37. 2024LLaVA-Zip: Adaptive Visual Token Compression with Intrinsic Image Informationcited by 3
  38. 2025ToFu: Visual Tokens Reduction via Fusion for Multi-modal, Multi-patch, Multi-image Taskcited by 3

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token
  2. 2024Matryoshka Query Transformer for Large Vision-Language Models
  3. 2025TrimTokenator-LC: Towards Adaptive Visual Token Pruning for Large Multimodal Models with Long Contexts
  4. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Models
  5. 2024A Concept-Based Explainability Framework for Large Multimodal Models
  6. 2024MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model
  7. 2024Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation
  8. 2025Learning Compact Vision Tokens for Efficient Large Multimodal Models