# A Parametric Memory Head for Continual Generative Retrieval

arXiv: https://arxiv.org/abs/2604.23388 · HTML: https://mlsanity.com/paper/2604.23388/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 25 Apr 2026.

## Builds on (most cited first, 30 of 31)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2010.08191/) (2020)
- [On Tiny Episodic Memories in Continual Learning](https://mlsanity.com/paper/1902.10486/) (2019)
- [Transformer Memory as a Differentiable Search Index](https://mlsanity.com/paper/2202.06991/) (2022)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [Large Memory Layers with Product Keys](https://mlsanity.com/paper/1907.05242/) (2019)
- [A Closer Look at Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2203.17269/) (2022)
- [Scalable and Effective Generative Information Retrieval](https://mlsanity.com/paper/2311.09134/) (2023)
- [DSI++: Updating Transformer Memory with New Documents](https://mlsanity.com/paper/2212.09744/) (2022)
- [Semantic-Enhanced Differentiable Search Index Inspired by Learning Strategies](https://mlsanity.com/paper/2305.15115/) (2023)
- [Continual Learning for Generative Retrieval over Dynamic Corpora](https://mlsanity.com/paper/2308.14968/) (2023)
- [Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction](https://mlsanity.com/paper/2204.10641/) (2022)
- [TOME: A Two-stage Approach for Model-based Retrieval](https://mlsanity.com/paper/2305.11161/) (2023)
- [Continual Learning via Sparse Memory Finetuning](https://mlsanity.com/paper/2510.15103/) (2025)
- [Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decoding](https://mlsanity.com/paper/2404.14600/) (2024)
- [Rethinking Search: Making Domain Experts out of Dilettantes](https://mlsanity.com/paper/2105.02274/) (2021)
- [Multiview Identifiers Enhanced Generative Retrieval](https://mlsanity.com/paper/2305.16675/) (2023)
- [Listwise Generative Retrieval Models via a Sequential Learning Process](https://mlsanity.com/paper/2403.12499/) (2024)
- [CorpusBrain++: A Continual Generative Pre-Training Framework for Knowledge-Intensive Language Tasks](https://mlsanity.com/paper/2402.16767/) (2024)
- [Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval](https://mlsanity.com/paper/2504.05181/) (2025)
- [Generative Retrieval Meets Multi-Graded Relevance](https://mlsanity.com/paper/2409.18409/) (2024)
- [Large Product Key Memory for Pretrained Language Models](https://mlsanity.com/paper/2010.03881/) (2020)
- [Replication and Exploration of Generative Retrieval over Dynamic Corpora](https://mlsanity.com/paper/2504.17519/) (2025)
- [MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora](https://mlsanity.com/paper/2507.09924/) (2025)

## Built on by (most cited first, 1 of 1)

- [ICICLE: Expanding Retrieval with In-Context Documents](https://mlsanity.com/paper/2605.26902/) (2026)

## Abstract (arXiv)

Generative information retrieval (GenIR) consolidates retrieval into a single neural model that decodes document identifiers (docids) directly from queries. While this model-as-index paradigm offers architectural simplicity, it is poorly suited to dynamic document collections. Unlike modular systems, where indexes are easily updated, GenIR's knowledge is parametrically encoded in its weights; consequently, standard adaptation methods such as full and parameter-efficient fine-tuning can induce catastrophic forgetting. We show that sequential adaptation improves retrieval on newly added documents but substantially degrades performance on earlier slices, exposing a pronounced stability-plasticity trade-off. To address this, we propose post-adaptation memory tuning (PAMT), a memory-only stabilization stage that augments an adapted model with a modular parametric memory head (PMH). PAMT freezes the backbone and attaches a product-key memory with fixed addressing. During prefix-trie constrained decoding, decoder hidden states sparsely query PMH to produce residual corrections in hidden space; these corrections are mapped to score adjustments via the frozen output embedding matrix, computed only over trie-valid tokens. This guides docid generation while keeping routing and backbone parameters fixed. To limit cross-slice interference, PAMT updates only a fixed budget of memory values selected using decoding-time access statistics, prioritizing entries frequently activated by the current slice and rarely used in prior sessions. Experiments on MS MARCO and Natural Questions under sequential, disjoint corpus increments show that PAMT substantially improves retention on earlier slices with minimal impact on retrieval performance for newly added documents, while modifying only a sparse subset of memory values per session.