# Training Recurrent Neural Networks by Sequential Least Squares and the Alternating Direction Method of Multipliers

arXiv: https://arxiv.org/abs/2112.15348 · HTML: https://mlsanity.com/paper/2112.15348/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 44 references that are in the map.
- First posted on arXiv 31 Dec 2021.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Learning Stable Deep Dynamics Models](https://mlsanity.com/paper/2001.06116/) (2020)
- [Global Convergence of ADMM in Nonconvex Nonsmooth Optimization](https://mlsanity.com/paper/1511.06324/) (2015)
- [Convergence Analysis of Alternating Direction Method of Multipliers for a Family of Nonconvex Problems](https://mlsanity.com/paper/1410.1390/) (2014)
- [Training Neural Networks Without Gradients: A Scalable ADMM Approach](https://mlsanity.com/paper/1605.02026/) (2016)
- [Nonlinear state-space identification using deep encoder networks](https://mlsanity.com/paper/2012.07697/) (2020)
- [diffGrad: An Optimization Method for Convolutional Neural Networks](https://mlsanity.com/paper/1909.11015/) (2019)
- [Gradient Sampling Methods for Nonsmooth Optimization](https://mlsanity.com/paper/1804.11003/) (2018)
- [Douglas-Rachford splitting and ADMM for nonconvex optimization: tight convergence results](https://mlsanity.com/paper/1709.05747/) (2017)
- [On the smoothness of nonlinear system identification](https://mlsanity.com/paper/1905.00820/) (2019)
- [E-RNN: Design Optimization for Efficient Recurrent Neural Networks in FPGAs](https://mlsanity.com/paper/1812.07106/) (2018)
- [Recurrent Neural Network Training with Convex Loss and Regularization Functions by Extended Kalman Filtering](https://mlsanity.com/paper/2111.02673/) (2021)
- [A Simple Effective Heuristic for Embedded Mixed-Integer Quadratic Programming](https://mlsanity.com/paper/1509.08416/) (2015)

## Built on by (most cited first, 8 of 8)

- [An L-BFGS-B approach for linear and nonlinear system identification under ℓ1 and group-Lasso regularization](https://mlsanity.com/paper/2403.03827/) (2024)
- [Enhanced Transformer architecture for in-context learning of dynamical systems](https://mlsanity.com/paper/2410.03291/) (2024)
- [Regularized Gauss-Newton for Optimizing Overparameterized Neural Networks](https://mlsanity.com/paper/2404.14875/) (2024)
- [Manifold meta-learning for reduced-complexity neural system identification](https://mlsanity.com/paper/2504.11811/) (2025)
- [Efficient identification of linear, parameter-varying, and nonlinear systems with noise models](https://mlsanity.com/paper/2504.11982/) (2025)
- [Nonlinear sparse variational Bayesian learning based model predictive control with application to PEMFC temperature control](https://mlsanity.com/paper/2404.09519/) (2024)
- [ADMM-Based Training for Spiking Neural Networks](https://mlsanity.com/paper/2505.05527/) (2025)
- [A proximal augmented Lagrangian method for nonconvex optimization with equality and inequality constraints](https://mlsanity.com/paper/2509.02894/) (2025)

## Abstract (arXiv)

This paper proposes a novel algorithm for training recurrent neural network models of nonlinear dynamical systems from an input/output training dataset. Arbitrary convex and twice-differentiable loss functions and regularization terms are handled by sequential least squares and either a line-search (LS) or a trust-region method of Levenberg-Marquardt (LM) type for ensuring convergence. In addition, to handle non-smooth regularization terms such as ℓ1, ℓ0, and group-Lasso regularizers, as well as to impose possibly non-convex constraints such as integer and mixed-integer constraints, we combine sequential least squares with the alternating direction method of multipliers (ADMM). We call the resulting algorithm NAILS (nonconvex ADMM iterations and least squares) in the case line search (LS) is used, or NAILM if a trust-region method (LM) is employed instead. The training method, which is also applicable to feedforward neural networks as a special case, is tested in three nonlinear system identification problems.