mlsanityData to 14 Oct 2026

Training Recurrent Neural Networks by Sequential Least Squares and the Alternating Direction Method of Multipliers

arXiv 2112.15348 ↗First posted 31 Dec 2021cs.LG cs.SY eess.SYcited by 8

Not on the map. Why?

Abstract (arXiv)

This paper proposes a novel algorithm for training recurrent neural network models of nonlinear dynamical systems from an input/output training dataset. Arbitrary convex and twice-differentiable loss functions and regularization terms are handled by sequential least squares and either a line-search (LS) or a trust-region method of Levenberg-Marquardt (LM) type for ensuring convergence. In addition, to handle non-smooth regularization terms such as ℓ1, ℓ0, and group-Lasso regularizers, as well as to impose possibly non-convex constraints such as integer and mixed-integer constraints, we combine sequential least squares with the alternating direction method of multipliers (ADMM). We call the resulting algorithm NAILS (nonconvex ADMM iterations and least squares) in the case line search (LS) is used, or NAILM if a trust-region method (LM) is employed instead. The training method, which is also applicable to feedforward neural networks as a special case, is tested in three nonlinear system identification problems.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 14 of its 44 references that are in the map. First posted on arXiv 31 Dec 2021.

Builds on

14 of 44 references.

References · 14

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019On the Convergence of Adam and Beyondcited by 841
  3. 2020Learning Stable Deep Dynamics Modelscited by 111
  4. 2015Global Convergence of ADMM in Nonconvex Nonsmooth Optimizationcited by 98
  5. 2014Convergence Analysis of Alternating Direction Method of Multipliers for a Family of Nonconvex Problemscited by 64
  6. 2016Training Neural Networks Without Gradients: A Scalable ADMM Approachcited by 61
  7. 2020Nonlinear state-space identification using deep encoder networkscited by 30
  8. 2019diffGrad: An Optimization Method for Convolutional Neural Networkscited by 29
  9. 2018Gradient Sampling Methods for Nonsmooth Optimizationcited by 23
  10. 2017Douglas-Rachford splitting and ADMM for nonconvex optimization: tight convergence resultscited by 21
  11. 2019On the smoothness of nonlinear system identificationcited by 17
  12. 2018E-RNN: Design Optimization for Efficient Recurrent Neural Networks in FPGAscited by 8
2 more
  1. 2021Recurrent Neural Network Training with Convex Loss and Regularization Functions by Extended Kalman Filteringcited by 8
  2. 2015A Simple Effective Heuristic for Embedded Mixed-Integer Quadratic Programmingcited by 5

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 4.

8 papers.

  1. 2024An L-BFGS-B approach for linear and nonlinear system identification under ℓ1 and group-Lasso regularizationcited by 8
  2. 2024Enhanced Transformer architecture for in-context learning of dynamical systemscited by 7
  3. 2024Regularized Gauss-Newton for Optimizing Overparameterized Neural Networkscited by 3
  4. 2025Manifold meta-learning for reduced-complexity neural system identificationcited by 2
  5. 2025Efficient identification of linear, parameter-varying, and nonlinear systems with noise modelscited by 2
  6. 2024Nonlinear sparse variational Bayesian learning based model predictive control with application to PEMFC temperature controlcited by 0
  7. 2025ADMM-Based Training for Spiking Neural Networkscited by 0
  8. 2025A proximal augmented Lagrangian method for nonconvex optimization with equality and inequality constraintscited by 0