# Efficient Training of Deep Equilibrium Models

arXiv: https://arxiv.org/abs/2304.11663 · HTML: https://mlsanity.com/paper/2304.11663/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 14 references that are in the map.
- First posted on arXiv 23 Apr 2023.

## Builds on (most cited first, 6 of 6)

- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Multiscale Deep Equilibrium Models](https://mlsanity.com/paper/2006.08656/) (2020)
- [Deep Equilibrium Architectures for Inverse Problems in Imaging](https://mlsanity.com/paper/2102.07944/) (2021)
- [JFB: Jacobian-Free Backpropagation for Implicit Networks](https://mlsanity.com/paper/2103.12803/) (2021)
- [On Training Implicit Models](https://mlsanity.com/paper/2111.05177/) (2021)
- [Music Source Separation with Deep Equilibrium Models](https://mlsanity.com/paper/2110.06494/) (2021)

## Built on by (most cited first, 2 of 2)

- [Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approach](https://mlsanity.com/paper/2602.03297/) (2026)
- [Towards Looped Models Done Right, Part II: Rethinking at Fixed Points](https://mlsanity.com/paper/2610.06833/) (2026)

## Abstract (arXiv)

Deep equilibrium models (DEQs) have proven to be very powerful for learning data representations. The idea is to replace traditional (explicit) feedforward neural networks with an implicit fixed-point equation, which allows to decouple the forward and backward passes. In particular, training DEQ layers becomes very memory-efficient via the implicit function theorem. However, backpropagation through DEQ layers still requires solving an expensive Jacobian-based equation. In this paper, we introduce a simple but effective strategy to avoid this computational burden. Our method relies on the Jacobian approximation of Broyden's method after the forward pass to compute the gradients during the backward pass. Experiments show that simply re-using this approximation can significantly speed up the training while not causing any performance degradation.