mlsanityData to 7 Oct 2026

Meta-Learning Bidirectional Update Rules

arXiv 2104.04657 ↗First posted 10 Apr 2021cs.LG cs.NEcited by 10

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce a new type of generalized neural network where neurons and synapses maintain multiple states. We show that classical gradient-based backpropagation in neural networks can be seen as a special case of a two-state network where one state is used for activations and another for gradients, with update rules derived from the chain rule. In our generalized framework, networks have neither explicit notion of nor ever receive gradients. The synapses and neurons are updated using a bidirectional Hebb-style update rule parameterized by a shared low-dimensional "genome". We show that such genomes can be meta-learned from scratch, using either conventional optimization techniques, or evolutionary strategies, such as CMA-ES. Resulting update rules generalize to unseen tasks and train faster than gradient descent based optimizers for several standard computer vision and synthetic tasks.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 39 of its 69 references that are in the map. First posted on arXiv 10 Apr 2021.

Builds on

39 of 69 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  6. 2016HyperNetworkscited by 1,003
  7. 2016Learning to learn by gradient descent by gradient descentcited by 616
  8. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  9. 2017EMNIST: an extension of MNIST to handwritten letterscited by 212
  10. 2014Difference Target Propagationcited by 144
  11. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 137
  12. 2020Descending through a Crowded Valley - Benchmarking Deep Learning Optimizerscited by 118
27 more
  1. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 113
  2. 2017Learned Optimizers that Scale and Generalizecited by 109
  3. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  4. 2019Deep Learning without Weight Transportcited by 87
  5. 2019Putting An End to End-to-End: Gradient-Isolated Learning of Representationscited by 85
  6. 2018Differentiable plasticity: training plastic neural networks with backpropagationcited by 83
  7. 2014How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagationcited by 77
  8. 2015Towards Biologically Plausible Deep Learningcited by 66
  9. 2016Training Neural Networks Without Gradients: A Scalable ADMM Approachcited by 61
  10. 2020Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselvescited by 57
  11. 2020LoCo: Local Contrastive Representation Learningcited by 55
  12. 2020Meta Learning Backpropagation And Improving Itcited by 49
  13. 2020Meta-Learning through Hebbian Plasticity in Random Networkscited by 47
  14. 2019Metalearned Neural Memorycited by 44
  15. 2020Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticitycited by 39
  16. 2018Biologically-plausible learning algorithms can scale to large datasetscited by 37
  17. 2018Continual Reinforcement Learning with Complex Synapsescited by 30
  18. 2017Born to Learn: the Inspiration, Progress, and Future of Evolved Plastic Artificial Neural Networkscited by 29
  19. 2020Reverse engineering learned optimizers reveals known and novel mechanismscited by 19
  20. 2020GAIT-prop: A biologically plausible learning rule derived from backpropagation of errorcited by 15
  21. 2020Learning to Learn with Feedback and Local Plasticitycited by 11
  22. 2017Diffusion-based neuromodulation can eliminate catastrophic forgetting in simple neural networkscited by 9
  23. 2020Continual Learning with Deep Artificial Neuronscited by 6
  24. 2020ZORB: A Derivative-Free Backpropagation Algorithm for Neural Networkscited by 5
  25. 2018Bidirectional Learning for Robust Neural Networkscited by 3
  26. 2018Neuromodulated Learning in Deep Neural Networkscited by 3
  27. 2020MPLP: Learning a Message Passing Learning Protocolcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 2; 2024: 2; 2025: 1.

10 papers.

  1. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  2. 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itselfcited by 40
  3. 2022Meta-Learning Biologically Plausible Plasticity Rules with Random Feedback Pathwayscited by 14
  4. 2022A Simple Guard for Learned Optimizerscited by 9
  5. 2022Meta Mirror Descent: Optimiser Learning for Fast Convergencecited by 7
  6. 2023Metalearning Continual Learning Algorithmscited by 6
  7. 2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 6
  8. 2024Evolving Self-Assembling Neural Networks: From Spontaneous Activity to Experience-Dependent Learningcited by 4
  9. 2024Narrowing the Focus: Learned Optimizers for Pretrained Modelscited by 3
  10. 2023Federated Automatic Differentiationcited by 2