# Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings

arXiv: https://arxiv.org/abs/2505.24782 · HTML: https://mlsanity.com/paper/2505.24782/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2025.

## Built on by (most cited first, 13 of 13)

- [ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios](https://mlsanity.com/paper/2601.08620/) (2026)
- [ModernVBERT: Towards Smaller Visual Document Retrievers](https://mlsanity.com/paper/2510.01149/) (2025)
- [Should We Still Pretrain Encoders with Masked Language Modeling?](https://mlsanity.com/paper/2507.00994/) (2025)
- [Diffusion-Pretrained Dense and Contextual Embeddings](https://mlsanity.com/paper/2602.11151/) (2026)
- [Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrieval](https://mlsanity.com/paper/2604.10167/) (2026)
- [LMEB: Long-horizon Memory Embedding Benchmark](https://mlsanity.com/paper/2603.12572/) (2026)
- [Contrastive Learning with Narrative Twins for Modeling Story Salience](https://mlsanity.com/paper/2601.07765/) (2026)
- [LIR: The First Workshop on Late Interaction and Multi Vector Retrieval @ ECIR 2026](https://mlsanity.com/paper/2511.00444/) (2025)
- [Embedding-Based Context-Aware Reranker](https://mlsanity.com/paper/2510.13329/) (2025)
- [AttentionRetriever: Attention Layers are Secretly Long Document Retrievers](https://mlsanity.com/paper/2602.12278/) (2026)
- [To MRL or not to MRL: Text Embeddings are Robust to Truncation Without Matryoshka Learning, Except In Heavy Truncation Scenarios](https://mlsanity.com/paper/2605.16608/) (2026)
- [CMDR: Contextual Multimodal Document Retrieval](https://mlsanity.com/paper/2607.05927/) (2026)
- [Beyond the Final Embedding: Retrieving Memories from Token-Level Evidence](https://mlsanity.com/paper/2610.13277/) (2026)

## Abstract (arXiv)

A limitation of modern document retrieval embedding methods is that they typically encode passages (chunks) from the same documents independently, often overlooking crucial contextual information from the rest of the document that could greatly improve individual chunk representations. In this work, we introduce ConTEB (Context-aware Text Embedding Benchmark), a benchmark designed to evaluate retrieval models on their ability to leverage document-wide context. Our results show that state-of-the-art embedding models struggle in retrieval scenarios where context is required. To address this limitation, we propose InSeNT (In-sequence Negative Training), a novel contrastive post-training approach which combined with late chunking pooling enhances contextual representation learning while preserving computational efficiency. Our method significantly improves retrieval quality on ConTEB without sacrificing base model performance. We further find chunks embedded with our method are more robust to suboptimal chunking strategies and larger retrieval corpus sizes. We open-source all artifacts at https://github.com/illuin-tech/contextual-embeddings.