mlsanityData to 14 Oct 2026

Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings

arXiv 2506.04997 ↗First posted 5 Jun 2025cs.IR cs.CLcited by 26

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

Despite the strong performance of ColPali/ColQwen2 in Visualized Document Retrieval (VDR), it encodes each page into multiple patch-level embeddings and leads to excessive memory usage. This empirical study investigates methods to reduce patch embeddings per page at minimum performance degradation. We evaluate two token-reduction strategies: token pruning and token merging. Regarding token pruning, we surprisingly observe that a simple random strategy outperforms other sophisticated pruning methods, though still far from satisfactory. Further analysis reveals that pruning is inherently unsuitable for VDR as it requires removing certain page embeddings without query-specific information. Turning to token merging (more suitable for VDR), we search for the optimal combinations of merging strategy across three dimensions and develop Light-ColPali/ColQwen2. It maintains 98.2% of retrieval performance with only 11.8% of original memory usage, and preserves 94.6% effectiveness at 2.8% memory footprint. We expect our empirical findings and resulting Light-ColPali/ColQwen2 offer valuable insights and establish a competitive baseline for future research towards efficient VDR.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 32 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

22 of 32 references.

References · 22

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  4. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  5. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  6. 2022Token Merging: Your ViT But Fastercited by 1,021
  7. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  8. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
  9. 2021InfographicVQAcited by 437
  10. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 363
  11. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 294
  12. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
10 more
  1. 2024MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizationscited by 183
  2. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 179
  3. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 167
  4. 2024M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understandingcited by 111
  5. 2024Unifying Multimodal Retrieval via Document Screenshot Embeddingcited by 110
  6. 2022Towards Complex Document Understanding By Discrete Reasoningcited by 82
  7. 2024AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledgecited by 44
  8. 2024AKEW: Assessing Knowledge Editing in the Wildcited by 36
  9. 2024Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Poolingcited by 25
  10. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 22.

26 papers.

Sort:

By area:

  1. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 20
  2. 2026Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Frameworkcited by 10
  3. 2025Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understandingcited by 9
  4. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  5. 2026CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embeddingcited by 7
  6. 2026Structural Anchor Pruning: Training-Free Multi-Vector Compression for Visual Document Retrievalcited by 6
  7. 2025Hybrid-Vector Retrieval for Visually Rich Documents: Combining Single-Vector Efficiency and Multi-Vector Accuracycited by 4
  8. 2026Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representationscited by 4
  9. 2026HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answeringcited by 3
  10. 2026NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrievalcited by 2
  11. 2026MINER: Mining Multimodal Internal Representation for Efficient Retrievalcited by 2
  12. 2026Very Efficient Listwise Multimodal Reranking for Long Documentscited by 2
14 more
  1. 2026VikingMem: A Memory Base Management System for Stateful LLM-based Applicationscited by 2
  2. 2026MM-Matryoshka: Towards Budget-Elastic Visual Document Retrieval via a 2D Multimodal Matryoshka Training Frameworkcited by 2
  3. 2026Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrievalcited by 2
  4. 2025LIR: The First Workshop on Late Interaction and Multi Vector Retrieval @ ECIR 2026cited by 1
  5. 2026AnchorFold: A Focus-Then-Fold Framework via Recursive Attention Propagation for Efficient Multi-Vector Visual Document Retrievalcited by 1
  6. 2026XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understandingcited by 0
  7. 2026KoVRE: Training an Efficient Embedding Model for Korean Visual Document Retrievalcited by 0
  8. 2026DistilVDR: A Compact End-to-End Visual Document Retriever via Dual-Student Distillationcited by 0
  9. 2026Spatial Matryoshka Training for Multi-Granularity Visual Document Retrievalcited by 0
  10. 2026MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Modelscited by 0
  11. 2026MAGIC: Marginal-Guided Compression with Optimal Transport for Efficient Visual Document Retrievalcited by 0
  12. 2026AdaMerge: Tuning-Free Patch Compression for Multi-Vector Visual Document Retrievalcited by 0
  13. 2026ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transportcited by 0
  14. 2026ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Hierarchical Patch Compression for ColPali: Efficient Multi-Vector Document Retrieval with Dynamic Pruning and Quantization
  2. 2024ColPali: Efficient Document Retrieval with Vision Language Models
  3. 2025Reproducibility, Replicability, and Insights into Visual Document Retrieval with Late Interaction
  4. 2026EVIE: Evidence-Vector-Informed Embeddings for Visual Document Retrieval ↗
  5. 2025Beyond Patch Aggregation: 3-Pass Pyramid Indexing for Vision-Enhanced Document Retrieval
  6. 2026Inverting Multi-Vector Visual Document Indices
  7. 2026DocPC: Document-Level Visual Retrieval via Representative Page Composition
  8. 2026MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval