mlsanityData to 7 Oct 2026

Meta Learning Backpropagation And Improving It

arXiv 2012.14905 ↗First posted 29 Dec 2020cs.LG cs.AI cs.NEcited by 49

Not on the map. Why?

Abstract (arXiv)

Many concepts have been proposed for meta learning with neural networks (NNs), e.g., NNs that learn to reprogram fast weights, Hebbian plasticity, learned learning rules, and meta recurrent NNs. Our Variable Shared Meta Learning (VSML) unifies the above and demonstrates that simple weight-sharing and sparsity in an NN is sufficient to express powerful learning algorithms (LAs) in a reusable fashion. A simple implementation of VSML where the weights of a neural network are replaced by tiny LSTMs allows for implementing the backpropagation LA solely by running in forward-mode. It can even meta learn new LAs that differ from online backpropagation and generalize to datasets outside of the meta training distribution without explicit gradient calculation. Introspection reveals that our meta learned LAs learn through fast association in a way that is qualitatively different from gradient descent.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 14; 2023: 11; 2024: 9; 2025: 8; 2026: 2.

49 papers.

Sort:

By area:

  1. 2022Transformers learn in-context by gradient descentcited by 714
  2. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  3. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  4. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  5. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  6. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  7. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  8. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  9. 2022Discovered Policy Optimisationcited by 103
  10. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  11. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  12. 2024Language Agents as Optimizable Graphscited by 81
37 more
  1. 2025End-to-End Test-Time Training for Long Contextcited by 53
  2. 2022Discovering Evolution Strategies via Meta-Black-Box Optimizationcited by 44
  3. 2025Learning without training: The implicit dynamics of in-context learningcited by 41
  4. 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itselfcited by 40
  5. 2023Discovering Attention-Based Genetic Algorithms via Meta-Black-Box Optimizationcited by 33
  6. 2023In-Context Learning for Attention Scheme: from Single Softmax Regression to Multiple Softmax Regression via a Tensor Trickcited by 32
  7. 2023Brain-inspired learning in artificial neural networks: a reviewcited by 26
  8. 2021Introducing Symmetries to Black Box Meta Reinforcement Learningcited by 25
  9. 2024Universal Neural Functionalscited by 25
  10. 2025Continuous Thought Machinescited by 21
  11. 2024Discovering Temporally-Aware Reinforcement Learning Algorithmscited by 18
  12. 2022Eliminating Meta Optimization Through Self-Referential Meta Learningcited by 12
  13. 2023Learning to (Learn at Test Time)cited by 12
  14. 2021Meta-Learning Bidirectional Update Rulescited by 10
  15. 2023The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Computecited by 10
  16. 2022A Simple Guard for Learned Optimizerscited by 9
  17. 2022Goal-Conditioned Generators of Deep Policiescited by 9
  18. 2024Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lenscited by 7
  19. 2023Metalearning Continual Learning Algorithmscited by 6
  20. 2024FFCL: Forward-Forward Net with Cortical Loops, Training and Inference on Edge Without Backpropagationcited by 6
  21. 2024Benchmarking General-Purpose In-Context Learningcited by 6
  22. 2024JaxLife: An Open-Ended Agentic Simulatorcited by 6
  23. 2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 6
  24. 2021Evolving Decomposed Plasticity Rules for Information-Bottlenecked Meta-Learningcited by 5
  25. 2022Learning to Optimize with Dynamic Mode Decompositioncited by 5
  26. 2023Extending the Forward Forward Algorithmcited by 5
  27. 2025How Should We Meta-Learn Reinforcement Learning Algorithms?cited by 5
  28. 2022Learning One Abstract Bit at a Time Through Self-Invented Experiments Encoded as Neural Networkscited by 4
  29. 2024Structurally Flexible Neural Networks: Evolving the Building Blocks for General Agentscited by 4
  30. 2026Training AI Scientists to Replicate Researchcited by 4
  31. 2023Are LSTMs Good Few-Shot Learners?cited by 3
  32. 2023Learning Functional Transductioncited by 2
  33. 2025Discovering Quality-Diversity Algorithms via Meta-Black-Box Optimizationcited by 2
  34. 2022Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networkscited by 1
  35. 2025Bio-Inspired Plastic Neural Networks for Zero-Shot Out-of-Distribution Generalization in Complex Animal-Inspired Robotscited by 1
  36. 2025NetworkFF: Unified Layer Optimization in Forward-Only Neural Networkscited by 1
  37. 2026Online Task Adaptation via Self-Organisationcited by 0