# A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithms

arXiv: https://arxiv.org/abs/2211.12386 · HTML: https://mlsanity.com/paper/2211.12386/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 99 references that are in the map.
- First posted on arXiv 22 Nov 2022.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Hierarchical Graph Representation Learning with Differentiable Pooling](https://mlsanity.com/paper/1806.08804/) (2018)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Meta-Learning with Implicit Gradients](https://mlsanity.com/paper/1909.04630/) (2019)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Optimizing Millions of Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/1911.02590/) (2019)
- [FractalNet: Ultra-Deep Neural Networks without Residuals](https://mlsanity.com/paper/1605.07648/) (2016)
- [A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights](https://mlsanity.com/paper/1503.01243/) (2015)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Hierarchical Representations for Efficient Architecture Search](https://mlsanity.com/paper/1711.00436/) (2017)
- [Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution](https://mlsanity.com/paper/1804.09081/) (2018)
- [Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks](https://mlsanity.com/paper/2106.04537/) (2021)
- [A Variational Perspective on Accelerated Methods in Optimization](https://mlsanity.com/paper/1603.04245/) (2016)
- [On Empirical Comparisons of Optimizers for Deep Learning](https://mlsanity.com/paper/1910.05446/) (2019)
- [Learning to Branch](https://mlsanity.com/paper/1803.10150/) (2018)
- [Reviving and Improving Recurrent Back-Propagation](https://mlsanity.com/paper/1803.06396/) (2018)
- [Multi-level Residual Networks from Dynamical Systems View](https://mlsanity.com/paper/1710.10348/) (2017)
- [Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves](https://mlsanity.com/paper/2009.11243/) (2020)
- [AdaNet: Adaptive Structural Learning of Artificial Neural Networks](https://mlsanity.com/paper/1607.01097/) (2016)
- [Geometry-Aware Gradient Algorithms for Neural Architecture Search](https://mlsanity.com/paper/2004.07802/) (2020)
- [EigenGame: PCA as a Nash Equilibrium](https://mlsanity.com/paper/2010.00554/) (2020)

## Built on by (most cited first, 8 of 8)

- [Looped Transformers are Better at Learning Learning Algorithms](https://mlsanity.com/paper/2311.12424/) (2023)
- [Identifying Equivalent Training Dynamics](https://mlsanity.com/paper/2302.09160/) (2023)
- [GoRINNs: Godunov-Riemann Informed Neural Networks for Learning Hyperbolic Conservation Laws](https://mlsanity.com/paper/2410.22193/) (2024)
- [Implementation and (Inverse Modified) Error Analysis for implicitly-templated ODE-nets](https://mlsanity.com/paper/2303.17824/) (2023)
- [Leveraging Operator Learning to Accelerate Convergence of the Preconditioned Conjugate Gradient Method](https://mlsanity.com/paper/2508.00101/) (2025)
- [Tiny Recursive Control: Iterative Reasoning for Efficient Optimal Control](https://mlsanity.com/paper/2512.16824/) (2025)
- [Fredholm Neural Networks for inverse problems in elliptic PDEs](https://mlsanity.com/paper/2507.06038/) (2025)
- [Learning Contractive Integral Operators with Fredholm Integral Neural Operators](https://mlsanity.com/paper/2604.03034/) (2026)

## Abstract (arXiv)

Meta-learning of numerical algorithms for a given task consists of the data-driven identification and adaptation of an algorithmic structure and the associated hyperparameters. To limit the complexity of the meta-learning problem, neural architectures with a certain inductive bias towards favorable algorithmic structures can, and should, be used. We generalize our previously introduced Runge-Kutta neural network to a recursively recurrent neural network (R2N2) superstructure for the design of customized iterative algorithms. In contrast to off-the-shelf deep learning approaches, it features a distinct division into modules for generation of information and for the subsequent assembly of this information towards a solution. Local information in the form of a subspace is generated by subordinate, inner, iterations of recurrent function evaluations starting at the current outer iterate. The update to the next outer iterate is computed as a linear combination of these evaluations, reducing the residual in this space, and constitutes the output of the network. We demonstrate that regular training of the weight parameters inside the proposed superstructure on input/output data of various computational problem classes yields iterations similar to Krylov solvers for linear equation systems, Newton-Krylov solvers for nonlinear equation systems, and Runge-Kutta integrators for ordinary differential equations. Due to its modularity, the superstructure can be readily extended with functionalities needed to represent more general classes of iterative algorithms traditionally based on Taylor series expansions.