# LEMUR: Learned Multi-Vector Retrieval

arXiv: https://arxiv.org/abs/2601.21853 · HTML: https://mlsanity.com/paper/2601.21853/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 53 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 30 of 31)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [The Faiss library](https://mlsanity.com/paper/2401.08281/) (2024)
- [ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms](https://mlsanity.com/paper/1807.05614/) (2018)
- [PLAID: An Efficient Engine for Late Interaction Retrieval](https://mlsanity.com/paper/2205.09707/) (2022)
- [jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval](https://mlsanity.com/paper/2506.18902/) (2025)
- [Relevance-guided Supervision for OpenQA with ColBERT](https://mlsanity.com/paper/2007.00814/) (2020)
- [Distilling Dense Representations for Ranking using Tightly-Coupled Teachers](https://mlsanity.com/paper/2010.11386/) (2020)
- [Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering](https://mlsanity.com/paper/2309.17133/) (2023)
- [LFM2 Technical Report](https://mlsanity.com/paper/2511.23404/) (2025)
- [Rethinking the Role of Token Retrieval in Multi-Vector Retrieval](https://mlsanity.com/paper/2304.01982/) (2023)
- [Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model](https://mlsanity.com/paper/2507.05513/) (2025)
- [ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios](https://mlsanity.com/paper/2601.08620/) (2026)
- [Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retriever](https://mlsanity.com/paper/2408.16672/) (2024)
- [Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling](https://mlsanity.com/paper/2409.14683/) (2024)
- [CAGRA: Highly Parallel Graph Construction and Approximate Nearest Neighbor Search for GPUs](https://mlsanity.com/paper/2308.15136/) (2023)
- [ModernVBERT: Towards Smaller Visual Document Retrievers](https://mlsanity.com/paper/2510.01149/) (2025)
- [DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning](https://mlsanity.com/paper/2509.23883/) (2025)
- [WARP: An Efficient Engine for Multi-Vector Retrieval](https://mlsanity.com/paper/2501.17788/) (2025)
- [JaColBERTv2.5: Optimising Multi-Vector Retrievers to Create State-of-the-Art Japanese Retrievers with Constrained Resources](https://mlsanity.com/paper/2407.20750/) (2024)
- [Efficient Constant-Space Multi-Vector Retrieval](https://mlsanity.com/paper/2504.01818/) (2025)
- [DESSERT: An Efficient Algorithm for Vector Set Search with Vector Set Queries](https://mlsanity.com/paper/2210.15748/) (2022)
- [Fantastic (small) Retrievers and How to Train Them: mxbai-edge-colbert-v0 Tech Report](https://mlsanity.com/paper/2510.14880/) (2025)
- [A Reproducibility Study of PLAID](https://mlsanity.com/paper/2404.14989/) (2024)
- [CRISP: Clustering Multi-Vector Representations for Denoising and Pruning](https://mlsanity.com/paper/2505.11471/) (2025)

## Built on by (most cited first, 1 of 1)

- [IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answering](https://mlsanity.com/paper/2602.17687/) (2026)

## Abstract (arXiv)

Multi-vector representations generated by late interaction models, such as ColBERT, enable superior retrieval quality compared to single-vector representations in information retrieval applications. In multi-vector retrieval systems, both queries and documents are encoded using one embedding per token, and similarity between queries and documents is measured by the MaxSim similarity measure. However, the improved quality of multi-vector retrieval comes at the expense of significantly increased search latency. In this work, we introduce LEMUR, a simple yet efficient framework for multi-vector similarity search. LEMUR consists of two consecutive problem reductions: First, we formulate multi-vector similarity search as a supervised learning problem that can be solved using a one-hidden-layer neural network. Second, we reduce inference under this model to single-vector similarity search in its latent space, enabling the use of existing single-vector search indexes to accelerate retrieval. LEMUR is an order of magnitude faster than prior multi-vector similarity search methods. Our code is available at https://github.com/ejaasaari/lemur