# Improving Long-Context Retrieval with Multi-Prefix Embedding

arXiv: https://arxiv.org/abs/2606.23642 · HTML: https://mlsanity.com/paper/2606.23642/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jun 2026.

## Built on by (most cited first, 1 of 1)

- [Beyond the Final Embedding: Retrieving Memories from Token-Level Evidence](https://mlsanity.com/paper/2610.13277/) (2026)

## Abstract (arXiv)

Long-context retrieval exposes a tension: single-vector embeddings lose fine-grained detail, while token-level multi-vector methods incur prohibitive storage. We propose Multi-Prefix Embedding (MPE), which partitions a document into chunks separated by EOS tokens, encodes the full sequence in a single causal forward pass, and extracts one embedding at each prefix boundary. MPE retains cross-chunk context, enables chunk-level MaxSim matching, and trains with only document-level relevance labels. Experiments on MLDR-en, BrowseComp-Plus, and LongEmbed show that MPE is competitive with or outperforms single-vector, independent-chunk, and multi-vector baselines, while providing a natural source attribution mechanism for locating evidence chunks.