mlsanityData to 7 Oct 2026

Fine-Tuning Pre-trained Transformers into Decaying Fast Weights

arXiv 2210.04243 ↗First posted 9 Oct 2022cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Autoregressive Transformers are strong language models but incur O(T) complexity during per-token generation due to the self-attention mechanism. Recent work proposes kernel-based methods to approximate causal self-attention by replacing it with recurrent formulations with various update rules and feature maps to achieve O(1) time and memory complexity. We explore these approaches and find that they are unnecessarily complex, and propose a simple alternative - decaying fast weights - that runs fast on GPU, outperforms prior methods, and retains 99% of attention's performance for GPT-2. We also show competitive performance on WikiText-103 against more complex attention substitutes.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 9 Oct 2022.

Builds on

28 of 41 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2020Rethinking Attention with Performerscited by 1,584
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2017Mixed Precision Trainingcited by 1,135
  10. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  11. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  12. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
16 more
  1. 2021Random Feature Attentioncited by 350
  2. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  3. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  4. 2020The Hardware Lotterycited by 160
  5. 2021An Attention Free Transformercited by 111
  6. 2021Finetuning Pretrained Transformers into RNNscited by 94
  7. 2018Stable Recurrent Modelscited by 65
  8. 2020Learning Associative Inference Using Fast Weight Memorycited by 51
  9. 2021ABC: Attention with Bounded-memory Controlcited by 36
  10. 2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
  11. 2018Sparse Attentive Backtracking: Temporal CreditAssignment Through Remindingcited by 33
  12. 2019Single Headed Attention RNN: Stop Thinking With Your Headcited by 25
  13. 2021Simple Local Attentions Remain Competitive for Long-Context Taskscited by 25
  14. 2018Reversible Recurrent Neural Networkscited by 22
  15. 2016A recurrent neural network without chaoscited by 13
  16. 2017MinimalRNN: Toward More Interpretable and Trainable Recurrent Neural Networkscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 13; 2025: 8; 2026: 3.

26 papers.

Sort:

By area:

  1. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  2. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  3. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  4. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  5. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  6. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  7. 2024DiG: Scalable and Efficient Diffusion Models with Gated Linear Attentioncited by 56
  8. 2024LinFusion: 1 GPU, 1 Minute, 16K Imagecited by 42
  9. 2025Log-Linear Attentioncited by 39
  10. 2024LoLCATs: On Low-Rank Linearizing of Large Language Modelscited by 25
  11. 2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Mapcited by 20
  12. 2024ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attentioncited by 13
14 more
  1. 2024Linear Attention Sequence Parallelismcited by 8
  2. 2025Linear Transformers as VAR Models: Aligning Autoregressive Attention Mechanisms with Autoregressive Forecastingcited by 8
  3. 2024Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspectivecited by 6
  4. 2025Resona: Improving Context Copying in Linear Recurrence Models with Retrievalcited by 5
  5. 2025ReGLA: Refining Gated Linear Attentioncited by 4
  6. 2024WAVE: Weighted Autoregressive Varying Gate for Time Series Forecastingcited by 3
  7. 2026Attention to Mamba: A Recipe for Cross-Architecture Distillationcited by 3
  8. 2025Untangling Component Imbalance in Hybrid Linear Attention Conversion Methodscited by 2
  9. 2024Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexitycited by 1
  10. 2025What Matters in Linearizing Language Models? A Comparative Study of Architecture, Scale, and Task Adaptationcited by 1
  11. 2025FlashEVA: Accelerating LLM inference via Efficient Attentioncited by 1
  12. 2026OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attentioncited by 1
  13. 2025Conformal Transformations for Symmetric Power Transformerscited by 0
  14. 2026Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Corescited by 0