# Beyond the Final Embedding: Retrieving Memories from Token-Level Evidence

arXiv: https://arxiv.org/abs/2610.13277 · HTML: https://mlsanity.com/paper/2610.13277/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 39 of its 43 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 30 of 39)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [LightMem: Lightweight and Efficient Memory-Augmented Generation](https://mlsanity.com/paper/2510.18866/) (2025)
- [Memory OS of AI Agent](https://mlsanity.com/paper/2506.06326/) (2025)
- [EmbeddingGemma: Powerful and Lightweight Text Representations](https://mlsanity.com/paper/2509.20354/) (2025)
- [Deeper Text Understanding for IR with Contextual Neural Language Modeling](https://mlsanity.com/paper/1905.09217/) (2019)
- [On the Theoretical Limitations of Embedding-Based Retrieval](https://mlsanity.com/paper/2508.21038/) (2025)
- [PLAID: An Efficient Engine for Late Interaction Retrieval](https://mlsanity.com/paper/2205.09707/) (2022)
- [NV-Retriever: Improving text embedding models with effective hard-negative mining](https://mlsanity.com/paper/2407.15831/) (2024)
- [Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models](https://mlsanity.com/paper/2409.04701/) (2024)
- [ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval](https://mlsanity.com/paper/2505.17166/) (2025)
- [Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks](https://mlsanity.com/paper/2511.07025/) (2025)
- [MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction](https://mlsanity.com/paper/2509.18095/) (2025)
- [REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation](https://mlsanity.com/paper/2502.13270/) (2025)
- [Rethinking the Role of Token Retrieval in Multi-Vector Retrieval](https://mlsanity.com/paper/2304.01982/) (2023)
- [PeerQA: A Scientific Question Answering Dataset from Peer Reviews](https://mlsanity.com/paper/2502.13668/) (2025)
- [jina-embeddings-v5-text: Task-Targeted Embedding Distillation](https://mlsanity.com/paper/2602.15547/) (2026)
- [WARP: An Efficient Engine for Multi-Vector Retrieval](https://mlsanity.com/paper/2501.17788/) (2025)
- [Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings](https://mlsanity.com/paper/2505.24782/) (2025)
- [Efficient Constant-Space Multi-Vector Retrieval](https://mlsanity.com/paper/2504.01818/) (2025)

## Abstract (arXiv)

Memory-augmented assistants retrieve evidence from stored conversations and documents to answer queries. Most embedding-based retrievers compare single-vector representations of queries and memories, even when relevance depends on a small part of the content. Such local evidence can be matched to queries using the contextual token representations already computed by the same embedding models. Our method uses sparse token matching to improve retrieval without additional training. Each query token retrieves a small set of matching tokens across candidate memories. The matching scores are aggregated for each memory and combined with whole-memory similarity to rank memories. An optional low-rank query adapter can be trained with supervision while keeping the encoder and stored memory representations fixed. Our method improves retrieval across eight benchmarks and multiple model families without additional training. Query adaptation further increases the gains to 13.3 and 14.3 percentage points in average Recall@10 and NDCG@10 on Qwen3-Embedding-4B. These retrieval gains translate into higher question-answering accuracy on long-term memory benchmarks.