mlsanityData to 7 Oct 2026

Adaptive Semiparametric Language Models

arXiv 2102.02557 ↗First posted 4 Feb 2021cs.CLcited by 83

Not on the map. Why?

Abstract (arXiv)

We present a language model that combines a large parametric neural network (i.e., a transformer) with a non-parametric episodic memory component in an integrated architecture. Our model uses extended short-term context by caching local hidden states -- similar to transformer-XL -- and global long-term memory by retrieving a set of nearest neighbor tokens at each timestep. We design a gating function to adaptively combine multiple information sources to make a prediction. This mechanism allows the model to use either local context, short-term memory, or long-term memory (or any combination of them) on an ad hoc basis depending on the context. Experiments on word-based and character-based language modeling datasets demonstrate the efficacy of our proposed method compared to strong baselines.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 4 Feb 2021.

Builds on

30 of 41 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2020Reformer: The Efficient Transformercited by 1,658
  12. 2020Rethinking Attention with Performerscited by 1,584
18 more
  1. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  3. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  4. 2020Nearest Neighbor Machine Translationcited by 209
  5. 2019Episodic Memory in Lifelong Language Learningcited by 188
  6. 2020Answering Complex Open-Domain Questions with Multi-Hop Dense Retrievalcited by 160
  7. 2017Generating Sentences by Editing Prototypescited by 88
  8. 2016Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modelingcited by 87
  9. 2016Improving Neural Language Models with a Continuous Cachecited by 81
  10. 2020MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Modelscited by 74
  11. 2016Efficient softmax approximation for GPUscited by 67
  12. 2017Learning to Remember Rare Eventscited by 63
  13. 2017Dynamic Evaluation of Neural Sequence Modelscited by 53
  14. 2017Unbounded cache model for online language modeling with open vocabularycited by 37
  15. 2020BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QAcited by 36
  16. 2019Non-Parametric Adaptation for Neural Machine Translationcited by 34
  17. 2019Dynamic Evaluation of Transformer Language Modelscited by 23
  18. 2016Generalizing and Hybridizing Count-based and Neural Language Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 27; 2023: 26; 2024: 14; 2025: 4; 2026: 1.

83 papers.

Sort:

By area:

  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  2. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  3. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  4. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  5. 2022A Contrastive Framework for Neural Text Generationcited by 224
  6. 2022Memorizing Transformerscited by 219
  7. 2021Internet-Augmented Dialogue Generationcited by 217
  8. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  9. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  10. 2022A Review on Language Models as Knowledge Basescited by 156
  11. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  12. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
38 more
  1. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  2. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  3. 2023Retrieval meets Long Context Large Language Modelscited by 114
  4. 2022Training Language Models with Memory Augmentationcited by 112
  5. 2023Lift Yourself Up: Retrieval-augmented Text Generation with Self Memorycited by 103
  6. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  7. 2021Efficient Nearest Neighbor Language Modelscited by 93
  8. 2021Data Augmentation Approaches in Natural Language Processing: A Surveycited by 82
  9. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  10. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  11. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Studycited by 63
  12. 2022Neuro-Symbolic Language Modeling with Automaton-augmented Retrievalcited by 60
  13. 2023InstructRetro: Instruction Tuning post Retrieval-Augmented Pretrainingcited by 53
  14. 2022Nonparametric Masked Language Modelingcited by 39
  15. 2021Towards mental time travel: a hierarchical memory for reinforcement learning agentscited by 38
  16. 2022Retrieval-Augmented Transformer for Image Captioningcited by 38
  17. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  18. 2022Combining Modular Skills in Multitask Learningcited by 27
  19. 2024CMAT: A Multi-Agent Collaboration Tuning Framework for Enhancing Small Language Modelscited by 26
  20. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processingcited by 25
  21. 2024Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selectioncited by 25
  22. 2022Relational Memory Augmented Language Modelscited by 24
  23. 2022Decoupled Context Processing for Context Augmented Language Modelingcited by 24
  24. 2023Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Modelscited by 23
  25. 2023Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Modelscited by 23
  26. 2022Chunk-based Nearest Neighbor Machine Translationcited by 21
  27. 2023Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Viewscited by 21
  28. 2023Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrievalcited by 20
  29. 2022ChapterBreak: A Challenge Dataset for Long-Range Language Modelscited by 18
  30. 2022You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LMcited by 18
  31. 2021You Only Need One Model for Open-domain Question Answeringcited by 17
  32. 2023Rethinking Privacy in Machine Learning Pipelines from an Information Flow Control Perspectivecited by 16
  33. 2025Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Modelscited by 16
  34. 2024Memorization in deep learning: A surveycited by 14
  35. 2023kNN-Adapter: Efficient Domain Adaptation for Black-Box Language Modelscited by 13
  36. 2023Transformer Working Memory Enables Regular Language Reasoning and Natural Language Length Extrapolationcited by 13
  37. 2023Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memorycited by 12
  38. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11

50 most cited of 83 All 83 (JSON)