# Robust learning with implicit residual networks

arXiv: https://arxiv.org/abs/1905.10479 · HTML: https://mlsanity.com/paper/1905.10479/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2019.

## Built on by (most cited first, 6 of 6)

- [Implicit Normalizing Flows](https://mlsanity.com/paper/2103.09527/) (2021)
- [Deciphering the Projection Head: Representation Evaluation Self-supervised Learning](https://mlsanity.com/paper/2301.12189/) (2023)
- [IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Method](https://mlsanity.com/paper/2505.06889/) (2025)
- [Accuracy and Architecture Studies of Residual Neural Network solving Ordinary Differential Equations](https://mlsanity.com/paper/2101.03583/) (2021)
- [Dissipative residual layers for unsupervised implicit parameterization of data manifolds](https://mlsanity.com/paper/2210.07100/) (2022)
- [Stability of implicit neural networks for long-term forecasting in dynamical systems](https://mlsanity.com/paper/2305.17155/) (2023)

## Abstract (arXiv)

In this effort, we propose a new deep architecture utilizing residual blocks inspired by implicit discretization schemes. As opposed to the standard feed-forward networks, the outputs of the proposed implicit residual blocks are defined as the fixed points of the appropriately chosen nonlinear transformations. We show that this choice leads to the improved stability of both forward and backward propagations, has a favorable impact on the generalization power and allows to control the robustness of the network with only a few hyperparameters. In addition, the proposed reformulation of ResNet does not introduce new parameters and can potentially lead to a reduction in the number of required layers due to improved forward stability. Finally, we derive the memory-efficient training algorithm, propose a stochastic regularization technique and provide numerical results in support of our findings.