mlsanityData to 7 Oct 2026

ESPN: Memory-Efficient Multi-Vector Information Retrieval

arXiv 2312.05417 ↗First posted 9 Dec 2023cs.IR cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Recent advances in large language models have demonstrated remarkable effectiveness in information retrieval (IR) tasks. While many neural IR systems encode queries and documents into single-vector representations, multi-vector models elevate the retrieval quality by producing multi-vector representations and facilitating similarity searches at the granularity of individual tokens. However, these models significantly amplify memory and storage requirements for retrieval indices by an order of magnitude. This escalation in index size renders the scalability of multi-vector IR models progressively challenging due to their substantial memory demands. We introduce Embedding from Storage Pipelined Network (ESPN) where we offload the entire re-ranking embedding tables to SSDs and reduce the memory requirements by 5-16x. We design a software prefetcher with hit rates exceeding 90%, improving SSD based retrieval up to 6.4x, and demonstrate that we can maintain near memory levels of query latency even for large query batch sizes.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 49 of its 81 references that are in the map. First posted on arXiv 9 Dec 2023.

Builds on

49 of 81 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
37 more
  1. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  2. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  3. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  4. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  5. 2020RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answeringcited by 410
  6. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  7. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  8. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  9. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
  10. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  11. 2019Multi-Stage Document Ranking with BERTcited by 200
  12. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  13. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  14. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 154
  15. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  16. 2021COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted Listcited by 113
  17. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
  18. 2017Fast Approximate Nearest Neighbor Search With The Navigating Spreading-out Graphcited by 103
  19. 2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 95
  20. 2021A Few Brief Notes on DeepImpact, COIL, and a Conceptual Framework for Information Retrieval Techniquescited by 69
  21. 2021Learning Passage Impacts for Inverted Indexescited by 68
  22. 2019Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrievalcited by 65
  23. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 47
  24. 2018Revisiting the Inverted Indices for Billion-Scale Approximate Nearest Neighborscited by 37
  25. 2021RecSSD: Near Data Processing for Solid State Drive Based Recommendation Inferencecited by 35
  26. 2021Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performancecited by 35
  27. 2021SPANN: Highly-efficient Billion-scale Approximate Nearest Neighbor Searchcited by 35
  28. 2020A Memory Efficient Baseline for Open Domain Question Answeringcited by 32
  29. 2021Pre-trained Language Model for Web-scale Retrieval in Baidu Searchcited by 31
  30. 2021Learning Discrete Representations via Constrained Clustering for Effective and Efficient Dense Retrievalcited by 27
  31. 2022Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reductioncited by 21
  32. 2022CITADEL: Conditional Token Interaction via Dynamic Lexical Routing for Efficient and Effective Multi-Vector Retrievalcited by 19
  33. 2021Wacky Weights in Learned Sparse Representations and the Revenge of Score-at-a-Time Query Evaluationcited by 15
  34. 2023An Introduction to the Compute Express Link (CXL) Interconnectcited by 9
  35. 2021SDR: Efficient Neural Re-ranking using Succinct Document Representationcited by 7
  36. 2022Dimension Reduction for Efficient Dense Retrieval via Conditional Autoencodercited by 6
  37. 2021Reducing Minor Page Fault Overheads through Enhanced Page Walkercited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  2. 2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrievalcited by 9
  3. 2026Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrievalcited by 6
  4. 2025Token-Driven GammaTune: Adaptive Calibration for Enhanced Speculative Decodingcited by 5
  5. 2025Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsitycited by 5
  6. 2026Beyond the Grid: Layout-Informed Multi-Vector Retrieval with Parsed Visual Document Representationscited by 3
  7. 2026MM-Matryoshka: Towards Budget-Elastic Visual Document Retrieval via a 2D Multimodal Matryoshka Training Frameworkcited by 2