# Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias

arXiv: https://arxiv.org/abs/2006.03824 · HTML: https://mlsanity.com/paper/2006.03824/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2020.

## Built on by (most cited first, 30 of 64)

- [Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](https://mlsanity.com/paper/2104.13386/) (2021)
- [Hebbian Deep Learning Without Feedback](https://mlsanity.com/paper/2209.11883/) (2022)
- [Holomorphic Equilibrium Propagation Computes Exact Gradients Through Finite Size Oscillations](https://mlsanity.com/paper/2209.00530/) (2022)
- [Energy-based learning algorithms for analog computing: a comparative study](https://mlsanity.com/paper/2312.15103/) (2023)
- [Brain-inspired Computational Intelligence via Predictive Coding](https://mlsanity.com/paper/2308.07870/) (2023)
- [Training an Ising Machine with Equilibrium Propagation](https://mlsanity.com/paper/2305.18321/) (2023)
- [Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation](https://mlsanity.com/paper/2205.03076/) (2022)
- [Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learning](https://mlsanity.com/paper/2206.02629/) (2022)
- [A Theoretical Framework for Inference Learning](https://mlsanity.com/paper/2206.00164/) (2022)
- [Self-Contrastive Forward-Forward Algorithm](https://mlsanity.com/paper/2409.11593/) (2024)
- [Agnostic Physics-Driven Deep Learning](https://mlsanity.com/paper/2205.15021/) (2022)
- [A contrastive rule for meta-learning](https://mlsanity.com/paper/2104.01677/) (2021)
- [Dual Propagation: Accelerating Contrastive Hebbian Learning with Dyadic Neurons](https://mlsanity.com/paper/2302.01228/) (2023)
- [Training Convolutional Neural Networks with the Forward-Forward algorithm](https://mlsanity.com/paper/2312.14924/) (2023)
- [Training Dynamical Binary Neural Networks with Equilibrium Propagation](https://mlsanity.com/paper/2103.08953/) (2021)
- [Towards Biologically Plausible Convolutional Networks](https://mlsanity.com/paper/2106.13031/) (2021)
- [Equilibrium Propagation: the Quantum and the Thermal Cases](https://mlsanity.com/paper/2405.08467/) (2024)
- [Quantum Equilibrium Propagation: Gradient-Descent Training of Quantum Systems](https://mlsanity.com/paper/2406.00879/) (2024)
- [Improving equilibrium propagation without weight symmetry through Jacobian homeostasis](https://mlsanity.com/paper/2309.02214/) (2023)
- [Learning long range dependencies through time reversal symmetry breaking](https://mlsanity.com/paper/2506.05259/) (2025)
- [Towards the Training of Deeper Predictive Coding Neural Networks](https://mlsanity.com/paper/2506.23800/) (2025)
- [A universal approximation theorem for nonlinear resistive networks](https://mlsanity.com/paper/2312.15063/) (2023)
- [A Review of Neuroscience-Inspired Machine Learning](https://mlsanity.com/paper/2403.18929/) (2024)
- [Equilibrium Propagation for Learning in Lagrangian Dynamical Systems](https://mlsanity.com/paper/2505.07363/) (2025)
- [Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methods](https://mlsanity.com/paper/2101.03419/) (2021)
- [Sequence Learning Using Equilibrium Propagation](https://mlsanity.com/paper/2209.09626/) (2022)
- [Toward Lifelong Learning in Equilibrium Propagation: Sleep-like and Awake Rehearsal for Enhanced Stability](https://mlsanity.com/paper/2508.14081/) (2025)
- [Towards training digitally-tied analog blocks via hybrid gradient computation](https://mlsanity.com/paper/2409.03306/) (2024)
- [Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation](https://mlsanity.com/paper/2411.02001/) (2024)
- [Biologically-inspired neuronal adaptation improves learning in neural networks](https://mlsanity.com/paper/2204.14008/) (2022)

## Abstract (arXiv)

Equilibrium Propagation (EP) is a biologically-inspired algorithm for convergent RNNs with a local learning rule that comes with strong theoretical guarantees. The parameter updates of the neural network during the credit assignment phase have been shown mathematically to approach the gradients provided by Backpropagation Through Time (BPTT) when the network is infinitesimally nudged toward its target. In practice, however, training a network with the gradient estimates provided by EP does not scale to visual tasks harder than MNIST. In this work, we show that a bias in the gradient estimate of EP, inherent in the use of finite nudging, is responsible for this phenomenon and that cancelling it allows training deep ConvNets by EP. We show that this bias can be greatly reduced by using symmetric nudging (a positive nudging and a negative one). We also generalize previous EP equations to the case of cross-entropy loss (by opposition to squared error). As a result of these advances, we are able to achieve a test error of 11.7% on CIFAR-10 by EP, which approaches the one achieved by BPTT and provides a major improvement with respect to the standard EP approach with same-sign nudging that gives 86% test error. We also apply these techniques to train an architecture with asymmetric forward and backward connections, yielding a 13.2% test error. These results highlight EP as a compelling biologically-plausible approach to compute error gradients in deep neural networks.