# Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrieval

arXiv: https://arxiv.org/abs/2604.10167 · HTML: https://mlsanity.com/paper/2604.10167/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 11 Apr 2026.

## Builds on (most cited first, 25 of 25)

- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents](https://mlsanity.com/paper/2410.10594/) (2024)
- [VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks](https://mlsanity.com/paper/2410.05160/) (2024)
- [GME: Improving Universal Multimodal Retrieval by Multimodal LLMs](https://mlsanity.com/paper/2412.16855/) (2024)
- [VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents](https://mlsanity.com/paper/2507.04590/) (2025)
- [MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing](https://mlsanity.com/paper/2509.22186/) (2025)
- [Unifying Multimodal Retrieval via Document Screenshot Embedding](https://mlsanity.com/paper/2406.11251/) (2024)
- [jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval](https://mlsanity.com/paper/2506.18902/) (2025)
- [Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models](https://mlsanity.com/paper/2409.04701/) (2024)
- [ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval](https://mlsanity.com/paper/2505.17166/) (2025)
- [UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning](https://mlsanity.com/paper/2510.13515/) (2025)
- [MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction](https://mlsanity.com/paper/2509.18095/) (2025)
- [Rethinking the Role of Token Retrieval in Multi-Vector Retrieval](https://mlsanity.com/paper/2304.01982/) (2023)
- [Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling](https://mlsanity.com/paper/2409.14683/) (2024)
- [jina-reranker-v3: Last but Not Late Interaction for Listwise Document Reranking](https://mlsanity.com/paper/2509.25085/) (2025)
- [ModernVBERT: Towards Smaller Visual Document Retrievers](https://mlsanity.com/paper/2510.01149/) (2025)
- [DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning](https://mlsanity.com/paper/2509.23883/) (2025)
- [Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings](https://mlsanity.com/paper/2505.24782/) (2025)
- [Multi-Vector Retrieval as Sparse Alignment](https://mlsanity.com/paper/2211.01267/) (2022)
- [Lost in OCR Translation? Vision-Based Approaches to Robust Document Retrieval](https://mlsanity.com/paper/2505.05666/) (2025)
- [ESPN: Memory-Efficient Multi-Vector Information Retrieval](https://mlsanity.com/paper/2312.05417/) (2023)
- [Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval](https://mlsanity.com/paper/2602.19961/) (2026)
- [CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding](https://mlsanity.com/paper/2601.21262/) (2026)
- [ReinPool: Reinforcement Learning Pooling Multi-Vector Embeddings for Retrieval System](https://mlsanity.com/paper/2601.07125/) (2026)
- [Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representations](https://mlsanity.com/paper/2603.01666/) (2026)

## Built on by (most cited first, 8 of 8)

- [MM-Matryoshka: Towards Budget-Elastic Visual Document Retrieval via a 2D Multimodal Matryoshka Training Framework](https://mlsanity.com/paper/2606.07654/) (2026)
- [AnchorFold: A Focus-Then-Fold Framework via Recursive Attention Propagation for Efficient Multi-Vector Visual Document Retrieval](https://mlsanity.com/paper/2608.08732/) (2026)
- [EigenLI: Spectral Approximations to Late Interaction](https://mlsanity.com/paper/2609.07561/) (2026)
- [VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval](https://mlsanity.com/paper/2608.17889/) (2026)
- [Spatial Matryoshka Training for Multi-Granularity Visual Document Retrieval](https://mlsanity.com/paper/2608.29951/) (2026)
- [MAGIC: Marginal-Guided Compression with Optimal Transport for Efficient Visual Document Retrieval](https://mlsanity.com/paper/2609.21018/) (2026)
- [Simple Linear Corrections for Compression in Late Interaction](https://mlsanity.com/paper/2610.14196/) (2026)
- [VL-DocIR: A Benchmark for Vision-Based Long Document Information Retrieval](https://mlsanity.com/paper/2610.17057/) (2026)

## Abstract (arXiv)

Multi-vector models dominate Visual Document Retrieval (VDR) due to their fine-grained matching capabilities, but their high storage and computational costs present a major barrier to practical deployment. In this paper, we propose ColChunk, a plug-and-play framework that introduces multimodal late chunking to construct efficient, contextualized multi-vectors. Unlike existing pruning or fixed-token approaches, ColChunk employs hierarchical clustering on patch-level embeddings, fused with a 2D position prior to ensure spatial-semantic coherence. This adaptive grouping allows for a content-aware representation that preserves global context while drastically reducing the vector count. Evaluations across 24 VDR datasets demonstrate ColChunk achieves over a 90% reduction in storage requirements while simultaneously delivering a 9-point average improvement in nDCG@5 across representative single-vector models. ColChunk provides a practical solution for balancing retrieval accuracy and efficiency in visual document systems.