mlsanityData to 14 Oct 2026

Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory

arXiv 2303.01421 ↗First posted 2 Mar 2023cs.CL cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Semiparametric language models (LMs) have shown promise in various Natural Language Processing (NLP) tasks. However, they utilize non-parametric memory as static storage, which lacks learning capability and remains disconnected from the internal information flow of the parametric models, limiting scalability and efficiency. Based on recent interpretability theories of LMs, we reconceptualize the non-parametric memory represented by kNN-LM as a learnable Mixture-of-Neighbors Induction Memory (MoNIM), which synergizes the induction capabilities of attention heads with the memorization strength of feed-forward networks (FFN). By integrating into the model's information flow, MoNIM functions as an FFN-like bypass layer within the Transformer architecture, enabling effective learning of new knowledge. Extensive experiments demonstrate that MoNIM is a retentive and scalable continual learner in both data- and model-wise, enhancing the scalability and continual learning performance of semiparametric LMs.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 27 of its 44 references that are in the map. First posted on arXiv 2 Mar 2023.

Builds on

27 of 44 references.

References · 27

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2017Billion-scale similarity search with GPUscited by 2,497
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  7. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  8. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  9. 2022In-context Learning and Induction Headscited by 964
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  11. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  12. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
15 more
  1. 2022RealTime QA: What's the Answer Right Now?cited by 244
  2. 2020Nearest Neighbor Machine Translationcited by 209
  3. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
  4. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  5. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 170
  6. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 160
  7. 2022Fine-tuned Language Models are Continual Learnerscited by 156
  8. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 115
  9. 2021Efficient Nearest Neighbor Language Modelscited by 93
  10. 2021Adaptive Semiparametric Language Modelscited by 83
  11. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  12. 2022Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scalecited by 64
  13. 2022You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LMcited by 18
  14. 2022Adaptation Approaches for Nearest Neighbor Language Modelscited by 6
  15. 2022Nearest Neighbor Language Models for Stylistic Controllable Generationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 2; 2025: 2; 2026: 2.

12 papers.

  1. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  2. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  3. 2023Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulationcited by 30
  4. 2023Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Modelscited by 23
  5. 2023Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Modelscited by 13
  6. 2025Pre-training Limited Memory Language Models with Internal and External Knowledgecited by 12
  7. 2025Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansioncited by 8
  8. 2024Collaborative Evolving Strategy for Automatic Data-Centric Developmentcited by 6
  9. 2024A Learning Rate Path Switching Training Paradigm for Version Updates of Large Language Modelscited by 2
  10. 2026Measuring and Mitigating Post-hoc Rationalization in Reverse Chain-of-Thought Generationcited by 2
  11. 2023Heterogenous Memory Augmented Neural Networkscited by 0
  12. 2026Encode Errors: Representational Retrieval of In-Context Demonstrations for Multilingual Grammatical Error Correctioncited by 0