mlsanityData to 7 Oct 2026

Meta-Learning Fast Weight Language Models

arXiv 2212.02475 ↗First posted 5 Dec 2022cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Dynamic evaluation of language models (LMs) adapts model parameters at test time using gradient information from previous tokens and substantially improves LM performance. However, it requires over 3x more compute than standard inference. We present Fast Weight Layers (FWLs), a neural component that provides the benefits of dynamic evaluation much more efficiently by expressing gradient updates as linear attention. A key improvement over dynamic evaluation is that FWLs can also be applied at training time so the model learns to make good use of gradient updates. FWLs can easily be added on top of existing transformer models, require relatively little extra compute or memory to run, and significantly improve language modeling perplexity.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 21 of its 24 references that are in the map. First posted on arXiv 5 Dec 2022.

Builds on

21 of 24 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2016Layer Normalizationcited by 5,579
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  8. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  9. 2021Fast Model Editing at Scalecited by 491
  10. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  11. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  12. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
9 more
  1. 2017Meta Networkscited by 268
  2. 2022Transformer Quality in Linear Timecited by 189
  3. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  4. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
  5. 2016Efficient softmax approximation for GPUscited by 67
  6. 2017Dynamic Evaluation of Neural Sequence Modelscited by 53
  7. 2019Metalearned Neural Memorycited by 44
  8. 2019Dynamic Evaluation of Transformer Language Modelscited by 23
  9. 2021Reconsidering the Past: Optimizing Hidden States in Language Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 4; 2025: 5; 2026: 5.

20 papers.

Sort:

By area:

  1. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  2. 2025One-Minute Video Generation with Test-Time Trainingcited by 100
  3. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  4. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  5. 2025End-to-End Test-Time Training for Long Contextcited by 53
  6. 2023Meta-Learning Online Adaptation of Language Modelscited by 43
  7. 2023Compressed Context Memory For Online Language Model Interactioncited by 42
  8. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
  9. 2024Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Passcited by 26
  10. 2025Lattice: Learning to Efficiently Compress the Memorycited by 18
  11. 2023Trainable Transformer in Transformercited by 17
  12. 2023kNN-Adapter: Efficient Domain Adaptation for Black-Box Language Modelscited by 13
8 more
  1. 2023Learning to (Learn at Test Time)cited by 12
  2. 2023When Meta-Learning Meets Online and Continual Learning: A Surveycited by 12
  3. 2024Online Test-Time Adaptation of Spatial-Temporal Traffic Flow Forecastingcited by 11
  4. 2026Rethinking Personalization in Large Language Models at the Token Levelcited by 3
  5. 2026MDN: Parallelizing Stepwise Momentum for Delta Linear Attentioncited by 3
  6. 2026Reinforced Fast Weights with Next-Sequence Predictioncited by 1
  7. 2026FocuSFT: Bilevel Optimization for Dilution-Aware Long-Context Fine-Tuningcited by 0
  8. 2026GradLev: Token-Parallel Test-Time Training Via Costate Predictioncited by 0