MM-Matryoshka: Towards Budget-Elastic Visual Document Retrieval via a 2D Multimodal Matryoshka Training Framework
On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG
Abstract (arXiv)
Multi-vector visual document retrievers achieve strong fine-grained matching by representing each page with multiple vectors from deep Vision-Language Models (VLMs), but this design makes deployment expensive in both storage and computational overhead. Existing efficiency techniques usually optimize only part of this budget, leaving multimodal retrievers without a unified way to trade accuracy for both vector width and encoder depth. Therefore, we propose MM-Matryoshka, a 2D Matryoshka training framework for budget-elastic Visual Document Retrieval (VDR), enabling ColPali-style multi-vector retrieval elastic along both dimension and layer. At inference time, a single retriever can select a 2D selectable budget without training separate models for different budgets. Through comprehensive experiments across multiple representative backbones, we demonstrate that by retaining significantly higher quality than direct truncation baselines while substantially reducing storage and computational overhead, MM-Matryoshka can offer robust budget elasticity for efficient VDR.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 39 references that are in the map. First posted on arXiv 3 Jun 2026.
Builds on
33 of 39 references.
Closest ancestors · 3
Same direction, newest first
- 2026CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding29 Jan
- 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMs22 Dec
- 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks7 Oct
Other references · 30
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
- 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
- 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 136
- 2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 95
- 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 72
- 2025ViDoRe Benchmark V2: Raising the Bar for Visual Retrievalcited by 58
- 2025mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Datacited by 49
- 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 36
- 2023Rethinking the Role of Token Retrieval in Multi-Vector Retrievalcited by 28
- 2025Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddingscited by 26
- 2025Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Modelcited by 24
18 more
- 2022CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrievalcited by 19
- 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
- 2024Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensionscited by 16
- 2025SMEC: Rethinking Matryoshka Representation Learning for Retrieval Embedding Compressioncited by 13
- 2024Train Once, Deploy Anywhere: Matryoshka Representation Learning for Multimodal Recommendationcited by 9
- 2026Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Frameworkcited by 8
- 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
- 2026Multi-Vector Index Compression in Any Modalitycited by 8
- 2023ESPN: Memory-Efficient Multi-Vector Information Retrievalcited by 7
- 2024Starbucks-v2: Improved Training for 2D Matryoshka Embeddingscited by 7
- 20242D Matryoshka Training for Information Retrievalcited by 6
- 2026Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrievalcited by 6
- 2025Hybrid-Vector Retrieval for Visually Rich Documents: Combining Single-Vector Efficiency and Multi-Vector Accuracycited by 4
- 2025Hierarchical Patch Compression for ColPali: Efficient Multi-Vector Document Retrieval with Dynamic Pruning and Quantizationcited by 3
- 2025Incorporating Token Importance in Multi-Vector Retrievalcited by 3
- 2026Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representationscited by 3
- 2026MIPIC: Matryoshka Representation Learning via Self-Distilled Intra-Relational and Progressive Information Chainingcited by 2
- 2026m3BERT: A Modern, Multi-lingual, Matryoshka Bidirectional Encodercited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transport
- 2026Visual RAG Toolkit: Scaling Multi-Vector Visual Retrieval with Training-Free Pooling and Multi-Stage Search
- 2026Structural Anchor Pruning: Training-Free Multi-Vector Compression for Visual Document Retrieval
- 2026Inference-Free Multimodal Learned Sparse Retrieval for Production-Scale Visual Document Search
- 2026NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrieval
- 2026MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval
- 2025ModernVBERT: Towards Smaller Visual Document Retrievers
- 2024ColPali: Efficient Document Retrieval with Vision Language Models