# PyLate: Flexible Training and Retrieval for Late Interaction Models

arXiv: https://arxiv.org/abs/2508.03555 · HTML: https://mlsanity.com/paper/2508.03555/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Nomic Embed: Training a Reproducible Long Context Text Embedder](https://mlsanity.com/paper/2402.01613/) (2024)
- [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://mlsanity.com/paper/2407.19669/) (2024)
- [RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking](https://mlsanity.com/paper/2110.07367/) (2021)
- [Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup](https://mlsanity.com/paper/2101.06983/) (2021)
- [PLAID: An Efficient Engine for Late Interaction Retrieval](https://mlsanity.com/paper/2205.09707/) (2022)
- [LongEmbed: Extending Embedding Models for Long Context Retrieval](https://mlsanity.com/paper/2404.12096/) (2024)
- [Simplified Data Wrangling with ir_datasets](https://mlsanity.com/paper/2103.02280/) (2021)
- [Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retriever](https://mlsanity.com/paper/2408.16672/) (2024)
- [Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling](https://mlsanity.com/paper/2409.14683/) (2024)
- [JaColBERTv2.5: Optimising Multi-Vector Retrievers to Create State-of-the-Art Japanese Retrievers with Constrained Resources](https://mlsanity.com/paper/2407.20750/) (2024)
- [Efficient Document Ranking with Learnable Late Interactions](https://mlsanity.com/paper/2406.17968/) (2024)

## Built on by (most cited first, 18 of 18)

- [On the Theoretical Limitations of Embedding-Based Retrieval](https://mlsanity.com/paper/2508.21038/) (2025)
- [jina-reranker-v3: Last but Not Late Interaction for Listwise Document Reranking](https://mlsanity.com/paper/2509.25085/) (2025)
- [OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries](https://mlsanity.com/paper/2605.06235/) (2026)
- [DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search](https://mlsanity.com/paper/2607.27178/) (2026)
- [ColBERT-Zero: To Pre-train Or Not To Pre-train ColBERT models](https://mlsanity.com/paper/2602.16609/) (2026)
- [Simple Projection Variants Improve ColBERT Performance](https://mlsanity.com/paper/2510.12327/) (2025)
- [Playing DOOM with 1.3M Parameters: Specialized Small Models vs Large Language Models for Real-Time Game Control](https://mlsanity.com/paper/2604.07385/) (2026)
- [LIR: The First Workshop on Late Interaction and Multi Vector Retrieval @ ECIR 2026](https://mlsanity.com/paper/2511.00444/) (2025)
- [TurkColBERT: A Benchmark of Dense and Late-Interaction Models for Turkish Information Retrieval](https://mlsanity.com/paper/2511.16528/) (2025)
- [EigenLI: Spectral Approximations to Late Interaction](https://mlsanity.com/paper/2609.07561/) (2026)
- [MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation](https://mlsanity.com/paper/2602.21379/) (2026)
- [A Unified Model and Document Representation for On-Device Retrieval-Augmented Generation](https://mlsanity.com/paper/2604.14403/) (2026)
- [The Multilingual Curse at the Retrieval Layer: Evidence from Amharic](https://mlsanity.com/paper/2605.24556/) (2026)
- [PROTOCOL: Late Interaction Retrieval for Protein Homolog Search](https://mlsanity.com/paper/2605.29158/) (2026)
- [MoganColBERT-TR: A Late-Interaction Multi-Vector Retrieval Model for Turkish](https://mlsanity.com/paper/2608.26344/) (2026)
- [OntologyBench: Can Dense Retrieval Satisfy Structured Biomedical Constraints?](https://mlsanity.com/paper/2609.08174/) (2026)
- [BELXTR: Biomedical Entity Linking via Contextualized Token Retrieval](https://mlsanity.com/paper/2609.25859/) (2026)
- [Recipe-Matching, Not Equivalence](https://mlsanity.com/paper/2609.31927/) (2026)

## Abstract (arXiv)

Neural ranking has become a cornerstone of modern information retrieval. While single vector search remains the dominant paradigm, it suffers from the shortcoming of compressing all the information into a single vector. This compression leads to notable performance degradation in out-of-domain, long-context, and reasoning-intensive retrieval tasks. Multi-vector approaches pioneered by ColBERT aim to address these limitations by preserving individual token embeddings and computing similarity via the MaxSim operator. This architecture has demonstrated superior empirical advantages, including enhanced out-of-domain generalization, long-context handling, and performance in complex retrieval scenarios. Despite these compelling empirical results and clear theoretical advantages, the practical adoption and public availability of late interaction models remain low compared to their single-vector counterparts, primarily due to a lack of accessible and modular tools for training and experimenting with such models. To bridge this gap, we introduce PyLate, a streamlined library built on top of Sentence Transformers to support multi-vector architectures natively, inheriting its efficient training, advanced logging, and automated model card generation while requiring minimal code changes to code templates users are already familiar with. By offering multi-vector-specific features such as efficient indexes, PyLate aims to accelerate research and real-world application of late interaction models, thereby unlocking their full potential in modern IR systems. Finally, PyLate has already enabled the development of state-of-the-art models, including GTE-ModernColBERT and Reason-ModernColBERT, demonstrating its practical utility for both research and production environments.