mlsanityData to 7 Oct 2026

A Modern Self-Referential Weight Matrix That Learns to Modify Itself

arXiv 2202.05780 ↗First posted 11 Feb 2022cs.LGcited by 40

Not on the map. Why?

Abstract (arXiv)

The weight matrix (WM) of a neural network (NN) is its program. The programs of many traditional NNs are learned through gradient descent in some error function, then remain fixed. The WM of a self-referential NN, however, can keep rapidly modifying all of itself during runtime. In principle, such NNs can meta-learn to learn, and meta-meta-learn to meta-learn to learn, and so on, in the sense of recursive self-improvement. While NN architectures potentially capable of implementing such behaviour have been proposed since the '90s, there have been few if any practical studies. Here we revisit such NNs, building upon recent successes of fast weight programmers and closely related linear Transformers. We propose a scalable self-referential WM (SRWM) that learns to use outer products and the delta update rule to modify itself. We evaluate our SRWM in supervised few-shot learning and in multi-task reinforcement learning with procedurally generated game environments. Our experiments demonstrate both practical applicability and competitive performance of the proposed SRWM. Our code is public.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 37 of its 78 references that are in the map. First posted on arXiv 11 Feb 2022.

Builds on

37 of 78 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2016Matching Networks for One Shot Learningcited by 2,246
  5. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  6. 2020Rethinking Attention with Performerscited by 1,584
  7. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  9. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  10. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  11. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  12. 2018TADAM: Task dependent adaptive metric for improved few-shot learningcited by 408
25 more
  1. 2016Learning to reinforcement learncited by 369
  2. 2021Random Feature Attentioncited by 350
  3. 2017Meta Networkscited by 268
  4. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  5. 2014Efficient Object Localization Using Convolutional Networkscited by 226
  6. 2020Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learningcited by 179
  7. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  8. 2022The CLEAR Benchmark: Continual LEArning on Real-World Imagerycited by 92
  9. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
  10. 2018Differentiable plasticity: training plastic neural networks with backpropagationcited by 83
  11. 2017Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithmcited by 65
  12. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  13. 2022HyperTransformer: Model Generation for Supervised and Semi-Supervised Few-Shot Learningcited by 55
  14. 2017Dynamic Evaluation of Neural Sequence Modelscited by 53
  15. 2019Torchmeta: A Meta-Learning library for PyTorchcited by 52
  16. 2015Increasing the Action Gap: New Operators for Reinforcement Learningcited by 49
  17. 2020Meta Learning Backpropagation And Improving Itcited by 49
  18. 2020Meta-Learning through Hebbian Plasticity in Random Networkscited by 47
  19. 2019Metalearned Neural Memorycited by 44
  20. 2019TorchBeast: A PyTorch Platform for Distributed RLcited by 41
  21. 2020Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticitycited by 39
  22. 2018Metalearning with Hebbian Fast Weightscited by 15
  23. 2020Addressing Catastrophic Forgetting in Few-Shot Problemscited by 12
  24. 2021Meta-Learning Bidirectional Update Rulescited by 10
  25. 2020Adaptive Reinforcement Learning through Evolving Self-Modifying Neural Networkscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 9; 2024: 7; 2025: 10; 2026: 8.

40 papers.

Sort:

By area:

  1. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  2. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  3. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  4. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  5. 2024Language Agents as Optimizable Graphscited by 81
  6. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  7. 2025Self-Adapting Language Modelscited by 71
  8. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  9. 2023Prompt Engineering a Prompt Engineercited by 53
  10. 2025End-to-End Test-Time Training for Long Contextcited by 53
  11. 2025Language Self-Play For Data-Free Trainingcited by 42
  12. 2023Brain-inspired learning in artificial neural networks: a reviewcited by 26
28 more
  1. 2023Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensionscited by 20
  2. 2022Learning to Generalize with Object-centric Agents in the Open World Survival Game Craftercited by 13
  3. 2022Eliminating Meta Optimization Through Self-Referential Meta Learningcited by 12
  4. 2023Deceptive Alignment Monitoringcited by 12
  5. 2025Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformerscited by 12
  6. 2026Memory Caching: RNNs with Growing Memorycited by 12
  7. 2023The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Computecited by 10
  8. 2024Real-Time Recurrent Learning using Trace Units in Reinforcement Learningcited by 10
  9. 2022Goal-Conditioned Generators of Deep Policiescited by 9
  10. 2022Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rulescited by 9
  11. 2023Metalearning Continual Learning Algorithmscited by 6
  12. 2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 6
  13. 2023Arbitrary Order Meta-Learning with Simple Population-Based Evolutioncited by 4
  14. 2024Overcoming classic challenges for artificial neural networks by providing incentives and practicecited by 3
  15. 2022Learning to learn online with neuromodulated synaptic plasticity in spiking neural networkscited by 2
  16. 2025Hypernetworks That Evolve Themselvescited by 2
  17. 2022Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networkscited by 1
  18. 2023Accelerating Neural Self-Improvement via Bootstrappingcited by 1
  19. 2025Recursive Dynamics in Fast-Weights Homeostatic Reentry Networks: Toward Reflective Intelligencecited by 1
  20. 2026Meta-Agent: From Task Descriptions to Verified Multi-Agent Systemscited by 1
  21. 2026Language Models Need Sleep: Learning to Self-Modify and Consolidate Memoriescited by 1
  22. 2026Proteus: Incremental Memory Activation for Long-Context Sequence Modelingcited by 1
  23. 2025Meta-Learning Transformers to Improve In-Context Generalizationcited by 0
  24. 2025Continuous-Time Homeostatic Dynamics for Reentrant Inference Modelscited by 0
  25. 2026SOLAR: A Self-Optimizing Open-Ended Autonomous Agent for Lifelong Learning and Continual Adaptationcited by 0
  26. 2026Path-dependent Discrete Amortized Inferencecited by 0
  27. 2026Canonical locks that encode part-whole hierarchiescited by 0
  28. 2026VisionHOPE: Visual Backbones as Self-Modifying Learning Systemscited by 0