# Recurrent Neural Network Training with Convex Loss and Regularization Functions by Extended Kalman Filtering

arXiv: https://arxiv.org/abs/2111.02673 · HTML: https://mlsanity.com/paper/2111.02673/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 42 references that are in the map.
- First posted on arXiv 4 Nov 2021.

## Builds on (most cited first, 8 of 8)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Model structures and fitting criteria for system identification with neural networks](https://mlsanity.com/paper/1911.13034/) (2019)
- [LSTM Neural Networks: Input to State Stability and Probabilistic Safety Verification](https://mlsanity.com/paper/1912.04377/) (2019)
- [Neural Network Training as an Optimal Control Problem: An Augmented Lagrangian Approach](https://mlsanity.com/paper/2103.14343/) (2021)
- [Controlling the level of sparsity in MPC](https://mlsanity.com/paper/1401.1369/) (2014)
- [Reduction of the Number of Variables in Parametric Constrained Least-Squares Problems](https://mlsanity.com/paper/2012.10423/) (2020)

## Built on by (most cited first, 8 of 8)

- [Training Recurrent Neural Networks by Sequential Least Squares and the Alternating Direction Method of Multipliers](https://mlsanity.com/paper/2112.15348/) (2021)
- [An L-BFGS-B approach for linear and nonlinear system identification under ℓ1 and group-Lasso regularization](https://mlsanity.com/paper/2403.03827/) (2024)
- [AI-Aided Kalman Filters](https://mlsanity.com/paper/2410.12289/) (2024)
- [On the Dynamics of Learning Time-Aware Behavior with Recurrent Neural Networks](https://mlsanity.com/paper/2306.07125/) (2023)
- [An active learning method for solving competitive multi-agent decision-making and control problems](https://mlsanity.com/paper/2212.12561/) (2022)
- [Data-Based Design of Multi-Model Inferential Sensors](https://mlsanity.com/paper/2308.02872/) (2023)
- [IRNN: Innovation-driven Recurrent Neural Network for Time-Series Data Modeling and Prediction](https://mlsanity.com/paper/2505.05916/) (2025)
- [A proximal augmented Lagrangian method for nonconvex optimization with equality and inequality constraints](https://mlsanity.com/paper/2509.02894/) (2025)

## Abstract (arXiv)

This paper investigates the use of extended Kalman filtering to train recurrent neural networks with rather general convex loss functions and regularization terms on the network parameters, including ℓ1-regularization. We show that the learning method is competitive with respect to stochastic gradient descent in a nonlinear system identification benchmark and in training a linear system with binary outputs. We also explore the use of the algorithm in data-driven nonlinear model predictive control and its relation with disturbance models for offset-free closed-loop tracking.