# Progressive Memory Banks for Incremental Domain Adaptation

arXiv: https://arxiv.org/abs/1811.00239 · HTML: https://mlsanity.com/paper/1811.00239/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2018.

## Built on by (most cited first, 12 of 12)

- [Distilled Replay: Overcoming Forgetting through Synthetic Samples](https://mlsanity.com/paper/2103.15851/) (2021)
- [Continual Learning for Recurrent Neural Networks: an Empirical Evaluation](https://mlsanity.com/paper/2103.07492/) (2021)
- [Memory-Augmented Non-Local Attention for Video Super-Resolution](https://mlsanity.com/paper/2108.11048/) (2021)
- [Studying Catastrophic Forgetting in Neural Ranking Models](https://mlsanity.com/paper/2101.06984/) (2021)
- [Knowledge Sharing via Domain Adaptation in Customs Fraud Detection](https://mlsanity.com/paper/2201.06759/) (2022)
- [CLEO: Continual Learning of Evolving Ontologies](https://mlsanity.com/paper/2407.08411/) (2024)
- [TokMem: One-Token Procedural Memory for Large Language Models](https://mlsanity.com/paper/2510.00444/) (2025)
- [Continual Learning of Long Topic Sequences in Neural Information Retrieval](https://mlsanity.com/paper/2201.03356/) (2022)
- [HOP to the Next Tasks and Domains for Continual Learning in NLP](https://mlsanity.com/paper/2402.18449/) (2024)
- [Smaller Is Better: An Analysis of Instance Quantity/Quality Trade-off in Rehearsal-based Continual Learning](https://mlsanity.com/paper/2105.14106/) (2021)
- [Adaptable Text Matching via Meta-Weight Regulator](https://mlsanity.com/paper/2204.12668/) (2022)
- [Mitigating Catastrophic Forgetting in Long Short-Term Memory Networks](https://mlsanity.com/paper/2305.17244/) (2023)

## Abstract (arXiv)

This paper addresses the problem of incremental domain adaptation (IDA) in natural language processing (NLP). We assume each domain comes one after another, and that we could only access data in the current domain. The goal of IDA is to build a unified model performing well on all the domains that we have encountered. We adopt the recurrent neural network (RNN) widely used in NLP, but augment it with a directly parameterized memory bank, which is retrieved by an attention mechanism at each step of RNN transition. The memory bank provides a natural way of IDA: when adapting our model to a new domain, we progressively add new slots to the memory bank, which increases the number of parameters, and thus the model capacity. We learn the new memory slots and fine-tune existing parameters by back-propagation. Experimental results show that our approach achieves significantly better performance than fine-tuning alone. Compared with expanding hidden states, our approach is more robust for old domains, shown by both empirical and theoretical results. Our model also outperforms previous work of IDA including elastic weight consolidation and progressive neural networks in the experiments.