mlsanityData to 7 Oct 2026

ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transport

arXiv 2609.34899 ↗First posted 28 Sep 2026cs.IR cs.CL cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

Multi-vector retrievers built on vision-language models lead visual document retrieval (VDR), but they run a multi-billion-parameter query encoder on every search. Distilling this encoder into a small student that queries the teacher's existing index would remove the bottleneck. The standard recipe, however, matches the teacher's MaxSim scores and so requires encoding and caching every training page, which can reach terabytes of page tokens. NanoVDR avoids pages entirely by training on the teacher's query embeddings alone, but only for single-vector retrievers. We present ColNanoVDR, to our knowledge the first framework to bring this document-free distillation to multi-vector VDR. Its objective, OTW (Optimal Transport with Learned Weights), aligns the student's query tokens with the teacher's by entropic optimal transport, with a learned weight for each student token, and needs no correspondence between the two tokenizations. We prove that the resulting alignment cost bounds the MaxSim score difference on every page. Distilled from five state-of-the-art teachers, the 149M text-only students retain about 95% of their teachers' NDCG@5 on ViDoRe v1-v3 while encoding queries up to 26x faster. Under identical training, OTW matches score distillation while encoding no page and reading 12.6x less cached teacher data.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 47 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

32 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrieval13 Mar
  2. 2026Structural Anchor Pruning: Training-Free Multi-Vector Compression for Visual Document Retrieval27 Jan

Other references · 30

  1. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  2. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  3. 2018Computational Optimal Transportcited by 765
  4. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  5. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  6. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  7. 2020Wasserstein Contrastive Representation Distillationcited by 63
  8. 2025ViDoRe Benchmark V2: Raising the Bar for Visual Retrievalcited by 58
  9. 2018Differential Properties of Sinkhorn Approximation for Learning with Wasserstein Distancecited by 56
  10. 2020Distilling Dense Representations for Ranking using Tightly-Coupled Teacherscited by 52
  11. 2024Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Modelscited by 35
  12. 2025Seq vs Seq: An Open Suite of Paired Encoders and Decoderscited by 34
18 more
  1. 2024MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encodingscited by 30
  2. 2022A Unified Framework for Implicit Sinkhorn Differentiationcited by 28
  3. 2025Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddingscited by 26
  4. 2024Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Poolingcited by 23
  5. 2026ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarioscited by 22
  6. 2022Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reductioncited by 21
  7. 2022Exploring Dual Encoder Architectures for Question Answeringcited by 19
  8. 2025ModernVBERT: Towards Smaller Visual Document Retrieverscited by 19
  9. 2024Sinkhorn Distance Minimization for Knowledge Distillationcited by 15
  10. 2024JaColBERTv2.5: Optimising Multi-Vector Retrievers to Create State-of-the-Art Japanese Retrievers with Constrained Resourcescited by 15
  11. 2023EmbedDistill: A Geometric Knowledge Distillation for Information Retrievalcited by 11
  12. 2025Fantastic (small) Retrievers and How to Train Them: mxbai-edge-colbert-v0 Tech Reportcited by 9
  13. 2021KNOT: Knowledge Distillation using Optimal Transport for Solving NLP Taskscited by 8
  14. 2023Query Encoder Distillation via Embedding Alignment is a Strong Baseline Method to Boost Dense Retriever Online Efficiencycited by 5
  15. 2025Incorporating Token Importance in Multi-Vector Retrievalcited by 3
  16. 2026A Voronoi Cell Formulation for Principled Token Pruning in Late-Interaction Retrieval Modelscited by 2
  17. 2023MCAD: Multi-teacher Cross-modal Alignment Distillation for efficient image-text retrievalcited by 1
  18. 2026Reproduction Beyond Benchmarks: ConstBERT and ColBERT-v2 Across Backends and Query Distributionscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning
  2. 2026AnchorFold: A Focus-Then-Fold Framework via Recursive Attention Propagation for Efficient Multi-Vector Visual Document Retrieval
  3. 2025SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models
  4. 2024ColPali: Efficient Document Retrieval with Vision Language Models
  5. 2026Inference-Free Multimodal Learned Sparse Retrieval for Production-Scale Visual Document Search
  6. 2026MM-Matryoshka: Towards Budget-Elastic Visual Document Retrieval via a 2D Multimodal Matryoshka Training Framework
  7. 2026Visual RAG Toolkit: Scaling Multi-Vector Visual Retrieval with Training-Free Pooling and Multi-Stage Search
  8. 2025QID: Efficient Query-Informed ViTs in Data-Scarce Regimes for OCR-free Visual Document Understanding