mlsanityData to 14 Oct 2026

A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithms

arXiv 2211.12386 ↗First posted 22 Nov 2022cs.LG cs.NA math.NAcited by 8

Not on the map. Why?

Abstract (arXiv)

Meta-learning of numerical algorithms for a given task consists of the data-driven identification and adaptation of an algorithmic structure and the associated hyperparameters. To limit the complexity of the meta-learning problem, neural architectures with a certain inductive bias towards favorable algorithmic structures can, and should, be used. We generalize our previously introduced Runge-Kutta neural network to a recursively recurrent neural network (R2N2) superstructure for the design of customized iterative algorithms. In contrast to off-the-shelf deep learning approaches, it features a distinct division into modules for generation of information and for the subsequent assembly of this information towards a solution. Local information in the form of a subspace is generated by subordinate, inner, iterations of recurrent function evaluations starting at the current outer iterate. The update to the next outer iterate is computed as a linear combination of these evaluations, reducing the residual in this space, and constitutes the output of the network. We demonstrate that regular training of the weight parameters inside the proposed superstructure on input/output data of various computational problem classes yields iterations similar to Krylov solvers for linear equation systems, Newton-Krylov solvers for nonlinear equation systems, and Runge-Kutta integrators for ordinary differential equations. Due to its modularity, the superstructure can be readily extended with functionalities needed to represent more general classes of iterative algorithms traditionally based on Taylor series expansions.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 40 of its 99 references that are in the map. First posted on arXiv 22 Nov 2022.

Builds on

40 of 99 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2018Neural Ordinary Differential Equationscited by 3,779
  5. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  6. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  7. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
  8. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  9. 2018Hierarchical Graph Representation Learning with Differentiable Poolingcited by 787
  10. 2016Learning to learn by gradient descent by gradient descentcited by 620
  11. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 599
  12. 2019Meta-Learning with Implicit Gradientscited by 525
28 more
  1. 2019Differentiable Convex Optimization Layerscited by 465
  2. 2017Stable Architectures for Deep Neural Networkscited by 341
  3. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 304
  4. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  5. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 206
  6. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  7. 2017Hierarchical Representations for Efficient Architecture Searchcited by 191
  8. 2018Efficient Multi-objective Neural Architecture Search via Lamarckian Evolutioncited by 153
  9. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 117
  10. 2016A Variational Perspective on Accelerated Methods in Optimizationcited by 110
  11. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 103
  12. 2018Learning to Branchcited by 78
  13. 2018Reviving and Improving Recurrent Back-Propagationcited by 69
  14. 2017Multi-level Residual Networks from Dynamical Systems Viewcited by 59
  15. 2020Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselvescited by 58
  16. 2016AdaNet: Adaptive Structural Learning of Artificial Neural Networkscited by 54
  17. 2020Geometry-Aware Gradient Algorithms for Neural Architecture Searchcited by 40
  18. 2020EigenGame: PCA as a Nash Equilibriumcited by 39
  19. 2018Direct Runge-Kutta Discretization Achieves Accelerationcited by 21
  20. 2018A machine learning framework for data driven acceleration of computations of differential equationscited by 18
  21. 2019Towards modular and programmable architecture searchcited by 11
  22. 2021Neural Fixed-Point Acceleration for Convex Optimizationcited by 10
  23. 2019Partial observations and conservation laws: Grey-box modeling in biotechnology and optogeneticscited by 9
  24. 2021Personalized Algorithm Generation: A Case Study in Learning ODE Integratorscited by 9
  25. 2022Algorithmic (Semi-)Conjugacy via Koopman Operator Theorycited by 8
  26. 2021Learning Dynamics from Noisy Measurements using Deep Learning with a Runge-Kutta Constraintcited by 7
  27. 2016Optimal Deterministic Algorithm Generationcited by 5
  28. 2020Discovery of Governing Equations with Recursive Deep Neural Networkscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 1; 2025: 3; 2026: 1.

8 papers.

  1. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 149
  2. 2023Identifying Equivalent Training Dynamicscited by 9
  3. 2024GoRINNs: Godunov-Riemann Informed Neural Networks for Learning Hyperbolic Conservation Lawscited by 8
  4. 2023Implementation and (Inverse Modified) Error Analysis for implicitly-templated ODE-netscited by 6
  5. 2025Leveraging Operator Learning to Accelerate Convergence of the Preconditioned Conjugate Gradient Methodcited by 6
  6. 2025Tiny Recursive Control: Iterative Reasoning for Efficient Optimal Controlcited by 4
  7. 2025Fredholm Neural Networks for inverse problems in elliptic PDEscited by 1
  8. 2026Learning Contractive Integral Operators with Fredholm Integral Neural Operatorscited by 0