mlsanityData to 7 Oct 2026

Structural Anchor Pruning: Training-Free Multi-Vector Compression for Visual Document Retrieval

arXiv 2601.20107 ↗First posted 27 Jan 2026cs.CV cs.CL cs.IRcited by 6

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Recent Vision-Language Models (e.g., ColPali) enable fine-grained Visual Document Retrieval (VDR) but incur prohibitive multi-vector index storage overhead. Existing training-free pruning methods either rely on heuristic layer choices or degrade sharply under aggressive compression, leading prior work to argue that effective high-compression pruning requires query-dependent training. We challenge this view with Structural Anchor Pruning (SAP), a self-calibrating, training-free, query-agnostic index-time framework combining (i) Score Retention (SR), a white-box per-layer compression diagnostic; (ii) SR-guided window selection, which automatically locates the structural pruning region of any backbone with no per-model hyperparameters; and (iii) a visual in-degree centrality scorer that identifies anchor patches within that window. On ViDoRe v1/v2 across three architectures spanning 18, 28, and 36 backbone layers, SAP retains 93--96% of NDCG@5 on v1 and 88--90% on the harder v2 while pruning 90% of visual tokens; at 20× compression it retains 85--90% and 76--79% respectively. Our layer-resolved SR analysis reveals an Alignment-Aggregation Divergence: visual structure is preserved as a stable ``Structural Plateau'' within the backbone, while the final layers reshape it into a sparse, query-aligned form unsuitable for pruning. Probing the pre-retrieval base backbones shows that contrastive fine-tuning sharpens this boundary three- to eight-fold, explaining why final-layer methods fail.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 16 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

11 of 16 references.

References · 11

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  3. 2022Token Merging: Your ViT But Fastercited by 995
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  5. 2013A Theoretical Analysis of NDCG Type Ranking Measurescited by 119
  6. 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 72
  7. 2025ViDoRe Benchmark V2: Raising the Bar for Visual Retrievalcited by 58
  8. 2025Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddingscited by 26
  9. 2025Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Modelcited by 24
  10. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  11. 2025Hierarchical Patch Compression for ColPali: Efficient Multi-Vector Document Retrieval with Dynamic Pruning and Quantizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  2. 2026NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrievalcited by 2
  3. 2026Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrievalcited by 2
  4. 2026AnchorFold: A Focus-Then-Fold Framework via Recursive Attention Propagation for Efficient Multi-Vector Visual Document Retrievalcited by 1
  5. 2026Spatial Matryoshka Training for Multi-Granularity Visual Document Retrievalcited by 0
  6. 2026ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Inference-Free Multimodal Learned Sparse Retrieval for Production-Scale Visual Document Search
  2. 2026RidgeRank: Efficient Visual Document Reranking via Score Fusion and a Shallow Linear Readout
  3. 2026AnchorPrune: Relevance-Anchored Contextual Expansion for Visual Token Pruning
  4. 2024ColPali: Efficient Document Retrieval with Vision Language Models
  5. 2026ReAlign: Optimizing the Visual Document Retriever with Reasoning-Guided Fine-Grained Alignment
  6. 2025ModernVBERT: Towards Smaller Visual Document Retrievers
  7. 2026MM-Matryoshka: Towards Budget-Elastic Visual Document Retrieval via a 2D Multimodal Matryoshka Training Framework
  8. 2026MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval