mlsanityData to 7 Oct 2026

Rethinking the Role of Token Retrieval in Multi-Vector Retrieval

arXiv 2304.01982 ↗First posted 4 Apr 2023cs.CL cs.IRcited by 28

Not on the map. Why?

Abstract (arXiv)

Multi-vector retrieval models such as ColBERT [Khattab and Zaharia, 2020] allow token-level interactions between queries and documents, and hence achieve state of the art on many information retrieval benchmarks. However, their non-linear scoring function cannot be scaled to millions of documents, necessitating a three-stage process for inference: retrieving initial candidates via token retrieval, accessing all token vectors, and scoring the initial candidate documents. The non-linear scoring function is applied over all token vectors of each candidate document, making the inference process complicated and slow. In this paper, we aim to simplify the multi-vector retrieval by rethinking the role of token retrieval. We present XTR, ConteXtualized Token Retriever, which introduces a simple, yet novel, objective function that encourages the model to retrieve the most important document tokens first. The improvement to token retrieval allows XTR to rank candidates only using the retrieved tokens rather than all tokens in the document, and enables a newly designed scoring stage that is two-to-three orders of magnitude cheaper than that of ColBERT. On the popular BEIR benchmark, XTR advances the state-of-the-art by 2.8 nDCG@10 without any distillation. Detailed analysis confirms our decision to revisit the token retrieval stage, as XTR demonstrates much better recall of the token retrieval stage compared to ColBERT.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 44 of its 50 references that are in the map. First posted on arXiv 4 Apr 2023.

Builds on

44 of 50 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2017Billion-scale similarity search with GPUscited by 2,477
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  7. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  9. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  10. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  11. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  12. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
32 more
  1. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  2. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  3. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  4. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  5. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  6. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
  7. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  8. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  9. 2019Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoringcited by 164
  10. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  11. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  12. 2020Pre-training Tasks for Embedding-based Large-scale Retrievalcited by 153
  13. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  14. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  15. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  16. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  17. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
  18. 2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 95
  19. 2020Learning Dense Representations of Phrases at Scalecited by 91
  20. 2019Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Indexcited by 64
  21. 2021Salient Phrase Aware Dense Retrieval: Can a Dense Retriever Imitate a Sparse One?cited by 59
  22. 2021Domain-matched Pre-training Tasks for Dense Retrievalcited by 53
  23. 2022Questions Are All You Need to Train a Dense Passage Retrievercited by 47
  24. 2022Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languagescited by 47
  25. 2021Learning to Retrieve Passages without Supervisioncited by 46
  26. 2022Multi-View Document Representation Learning for Open-Domain Dense Retrievalcited by 41
  27. 2021Phrase Retrieval Learns Passage Retrieval, Toocited by 33
  28. 2022CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrievalcited by 19
  29. 2014Improved Asymmetric Locality Sensitive Hashing (ALSH) for Maximum Inner Product Search (MIPS)cited by 15
  30. 2021Shallow pooling for sparse labelscited by 13
  31. 2022Multi-Vector Retrieval as Sparse Alignmentcited by 12
  32. 2021On Approximate Nearest Neighbour Selection for Multi-Stage Dense Retrievalcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 5; 2026: 16.

28 papers.

Sort:

By area:

  1. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  2. 2025MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interactioncited by 36
  3. 2024Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Poolingcited by 23
  4. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  5. 2024Automated Knowledge Concept Annotation and Question Representation Learning for Knowledge Tracingcited by 11
  6. 2025Efficient Constant-Space Multi-Vector Retrievalcited by 10
  7. 2024Matchmaker: Self-Improving Large Language Model Programs for Schema Matchingcited by 9
  8. 2024Generative Retrieval as Multi-Vector Dense Retrievalcited by 8
  9. 2024ColBERT-XM: A Modular Multi-Vector Representation Model for Zero-Shot Multilingual Information Retrievalcited by 7
  10. 2026Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrievalcited by 6
  11. 2026Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representationscited by 3
  12. 2025Simple Projection Variants Improve ColBERT Performancecited by 2
16 more
  1. 2026MM-Matryoshka: Towards Budget-Elastic Visual Document Retrieval via a 2D Multimodal Matryoshka Training Frameworkcited by 2
  2. 2023Back to Basics: A Simple Recipe for Improving Out-of-Domain Retrieval in Dense Encoderscited by 1
  3. 2025LIR: The First Workshop on Late Interaction and Multi Vector Retrieval @ ECIR 2026cited by 1
  4. 2026LEMUR: Learned Multi-Vector Retrievalcited by 1
  5. 2026LLM-Oriented Information Retrieval: A Denoising-First Perspectivecited by 1
  6. 2026LiteTopK: Exploiting the Curse of Dimensionality for a Fused Indexer-TopK Kernel in Long-Context Sparse Attentioncited by 1
  7. 2026GEM: A Native Graph-based Index for Multi-Vector Retrievalcited by 0
  8. 2026PROTOCOL: Late Interaction Retrieval for Protein Homolog Searchcited by 0
  9. 2026No More K-means: Single-Stage Sparse Coding for Efficient Multi-Vector Retrievalcited by 0
  10. 2026ColBERTSaR: Sparsified ColBERT Index via Product Quantizationcited by 0
  11. 2026PLAID-PRF: Pseudo-Relevance Feedback with Centroid-like Tokens in PLAIDcited by 0
  12. 2026MoganColBERT-TR: A Late-Interaction Multi-Vector Retrieval Model for Turkishcited by 0
  13. 2026Spatial Matryoshka Training for Multi-Granularity Visual Document Retrievalcited by 0
  14. 2026Query-Aware Token Budgeting for Efficient Late-Interaction Visual Document Retrievalcited by 0
  15. 2026EigenLI: Spectral Approximations to Late Interactioncited by 0
  16. 2026BELXTR: Biomedical Entity Linking via Contextualized Token Retrievalcited by 0