mlsanityData to 7 Oct 2026

Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling

arXiv 2409.14683 ↗First posted 23 Sep 2024cs.IR cs.AI cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

Over the last few years, multi-vector retrieval methods, spearheaded by ColBERT, have become an increasingly popular approach to Neural IR. By storing representations at the token level rather than at the document level, these methods have demonstrated very strong retrieval performance, especially in out-of-domain settings. However, the storage and memory requirements necessary to store the large number of associated vectors remain an important drawback, hindering practical adoption. In this paper, we introduce a simple clustering-based token pooling approach to aggressively reduce the number of vectors that need to be stored. This method can reduce the space & memory footprint of ColBERT indexes by 50% with virtually no retrieval performance degradation. This method also allows for further reductions, reducing the vector count by 66%-to-75% , with degradation remaining below 5% on a vast majority of datasets. Importantly, this approach requires no architectural change nor query-time processing, and can be used as a simple drop-in during indexation with any ColBERT-like model.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 16 of its 39 references that are in the map. First posted on arXiv 23 Sep 2024.

Builds on

16 of 39 references.

References · 16

  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  2. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  3. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  4. 2022Token Merging: Your ViT But Fastercited by 995
  5. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  6. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  7. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  8. 2023Token Merging for Fast Stable Diffusioncited by 220
  9. 2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 95
  10. 2021Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performancecited by 35
  11. 2023Rethinking the Role of Token Retrieval in Multi-Vector Retrievalcited by 28
  12. 2022Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reductioncited by 21
4 more
  1. 2022MS-Shift: An Analysis of MS MARCO Distribution Shifts on Neural Retrievalcited by 10
  2. 2022Evaluating Interpolation and Extrapolation Performance of Neural Retrieval Modelscited by 9
  3. 2024Systematic Evaluation of Neural Retrieval Models on the Touch'e 2020 Argument Retrieval Subset of BEIRcited by 9
  4. 2022What company do words keep? Revisiting the distributional semantics of J.R. Firth & Zellig Harriscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 14.

23 papers.

Sort:

By area:

  1. 2025Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddingscited by 26
  2. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  3. 2025PyLate: Flexible Training and Retrieval for Late Interaction Modelscited by 17
  4. 2025Efficient Constant-Space Multi-Vector Retrievalcited by 10
  5. 2026Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Frameworkcited by 8
  6. 2026Multi-Vector Index Compression in Any Modalitycited by 8
  7. 2026Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrievalcited by 6
  8. 2025ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrievalcited by 5
  9. 2025Hybrid-Vector Retrieval for Visually Rich Documents: Combining Single-Vector Efficiency and Multi-Vector Accuracycited by 4
  10. 2025Simple Projection Variants Improve ColBERT Performancecited by 2
  11. 2026IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answeringcited by 2
  12. 2025LIR: The First Workshop on Late Interaction and Multi Vector Retrieval @ ECIR 2026cited by 1
11 more
  1. 2025SMART: Shot-Aware Multimodal Video Moment Retrieval with Audio-Enhanced MLLMcited by 1
  2. 2026LEMUR: Learned Multi-Vector Retrievalcited by 1
  3. 2026MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrievalcited by 1
  4. 2026AnchorFold: A Focus-Then-Fold Framework via Recursive Attention Propagation for Efficient Multi-Vector Visual Document Retrievalcited by 1
  5. 2026CMDR: Contextual Multimodal Document Retrievalcited by 0
  6. 2026H+ Embedding: Harmonizing Global and Token-Level Retrieval with Context-Dependent Phrasescited by 0
  7. 2026Query-Aware Token Budgeting for Efficient Late-Interaction Visual Document Retrievalcited by 0
  8. 2026EigenLI: Spectral Approximations to Late Interactioncited by 0
  9. 2026ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transportcited by 0
  10. 2026ResComEmb: Effective and Efficient Multimodal Embedding via Residual Homogeneity Compressioncited by 0
  11. 2026Inverting Multi-Vector Visual Document Indicescited by 0