mlsanityData to 7 Oct 2026

Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules

arXiv 2206.01649 ↗First posted 3 Jun 2022cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Neural ordinary differential equations (ODEs) have attracted much attention as continuous-time counterparts of deep residual neural networks (NNs), and numerous extensions for recurrent NNs have been proposed. Since the 1980s, ODEs have also been used to derive theoretical results for NN learning rules, e.g., the famous connection between Oja's rule and principal component analysis. Such rules are typically expressed as additive iterative update processes which have straightforward ODE counterparts. Here we introduce a novel combination of learning rules and Neural ODEs to build continuous-time sequence processing nets that learn to manipulate short-term memory in rapidly changing synaptic connections of other nets. This yields continuous-time counterparts of Fast Weight Programmers and linear Transformers. Our novel models outperform the best existing Neural Controlled Differential Equation based models on various time series classification tasks, while also addressing their fundamental scalability limitations. Our code is public.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 35 of its 85 references that are in the map. First posted on arXiv 3 Jun 2022.

Builds on

35 of 85 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2016Layer Normalizationcited by 5,579
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2016HyperNetworkscited by 1,003
  8. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  9. 2016Recurrent Neural Networks for Multivariate Time Series with Missing Valuescited by 548
  10. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 501
  11. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  12. 2017Stable Architectures for Deep Neural Networkscited by 340
23 more
  1. 2018The UEA multivariate time series classification archive, 2018cited by 330
  2. 2022On Neural Differential Equationscited by 264
  3. 2015Highway Networkscited by 257
  4. 2019GRU-ODE-Bayes: Continuous modeling of sporadically-observed time seriescited by 230
  5. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  6. 2020Neural Rough Differential Equations for Long Time Seriescited by 140
  7. 2019Language Modeling with Deep Transformerscited by 96
  8. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
  9. 2017Reversible Architectures for Arbitrarily Deep Residual Neural Networkscited by 69
  10. 2017Multi-level Residual Networks from Dynamical Systems Viewcited by 59
  11. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  12. 2021Neural Controlled Differential Equations for Online Prediction Taskscited by 46
  13. 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 45
  14. 2021Long Expressive Memory for Sequence Modelingcited by 41
  15. 2021Continuous-Time Model-Based Reinforcement Learningcited by 39
  16. 2017Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learningcited by 38
  17. 2020Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEscited by 32
  18. 2021Framing RNN as a kernel method: A neural ODE approachcited by 23
  19. 2020An Ode to an ODEcited by 17
  20. 2022Continuous-Time Meta-Learning with Forward Mode Differentiationcited by 12
  21. 2018NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equationscited by 7
  22. 2021Training and Generating Neural Networks in Compressed Weight Spacecited by 4
  23. 2019ODE-Inspired Analysis for the Biological Version of Oja's Rule in Solving Streaming PCAcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 3; 2025: 5; 2026: 3.

15 papers.

Sort:

By area:

  1. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  2. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  3. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  4. 2025End-to-End Test-Time Training for Long Contextcited by 53
  5. 2023Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensionscited by 20
  6. 2022Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rulescited by 9
  7. 2023Metalearning Continual Learning Algorithmscited by 6
  8. 2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 6
  9. 2024DualDynamics: Synergizing Implicit and Explicit Methods for Robust Irregular Time Series Analysiscited by 4
  10. 2022Learning to Control Rapidly Changing Synaptic Connections: An Alternative Type of Memory in Sequence Processing Artificial Neural Networkscited by 1
  11. 2026Proteus: Incremental Memory Activation for Long-Context Sequence Modelingcited by 1
  12. 2025Who invented deep residual learning?cited by 0
3 more
  1. 2025BioOSS: A Bio-Inspired Oscillatory State System with Spatio-Temporal Dynamicscited by 0
  2. 2026Integrating Mechanistic and Data-Driven Models for Neurological Disorders through Differentiable Programmingcited by 0
  3. 2026Online Neural Space Time Memory for Dynamic Novel View Synthesiscited by 0