# Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learning

arXiv: https://arxiv.org/abs/2206.02629 · HTML: https://mlsanity.com/paper/2206.02629/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 58 references that are in the map.
- First posted on arXiv 31 May 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [Difference Target Propagation](https://mlsanity.com/paper/1412.7525/) (2014)
- [Predictive Coding Approximates Backprop along Arbitrary Computation Graphs](https://mlsanity.com/paper/2006.04182/) (2020)
- [Predictive Coding: a Theoretical and Experimental Review](https://mlsanity.com/paper/2107.12979/) (2021)
- [The free energy principle for action and perception: A mathematical review](https://mlsanity.com/paper/1705.09156/) (2017)
- [Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias](https://mlsanity.com/paper/2006.03824/) (2020)
- [Training End-to-End Analog Neural Networks with Equilibrium Propagation](https://mlsanity.com/paper/2006.01981/) (2020)
- [A Theoretical Framework for Target Propagation](https://mlsanity.com/paper/2006.14331/) (2020)
- [Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures](https://mlsanity.com/paper/2006.12878/) (2020)
- [Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?](https://mlsanity.com/paper/2202.09467/) (2022)
- [Associative Memories via Predictive Coding](https://mlsanity.com/paper/2109.08063/) (2021)
- [Biologically Motivated Algorithms for Propagating Local Target Representations](https://mlsanity.com/paper/1805.11703/) (2018)
- [Learning on Arbitrary Graph Topologies via Predictive Coding](https://mlsanity.com/paper/2201.13180/) (2022)
- [Equivalence of Equilibrium Propagation and Recurrent Backpropagation](https://mlsanity.com/paper/1711.08416/) (2017)
- [Equilibrium Propagation with Continual Weight Updates](https://mlsanity.com/paper/2005.04168/) (2020)
- [Reverse Differentiation via Predictive Coding](https://mlsanity.com/paper/2103.04689/) (2021)
- [Early Inference in Energy-Based Models Approximates Back-Propagation](https://mlsanity.com/paper/1510.02777/) (2015)
- [Relaxing the Constraints on Predictive Coding Models](https://mlsanity.com/paper/2010.01047/) (2020)
- [Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain](https://mlsanity.com/paper/2009.05359/) (2020)
- [Habitual and Reflective Control in Hierarchical Predictive Coding](https://mlsanity.com/paper/2109.00866/) (2021)
- [PredProp: Bidirectional Stochastic Optimization with Precision Weighted Predictive Coding](https://mlsanity.com/paper/2111.08792/) (2021)

## Built on by (most cited first, 26 of 26)

- [Energy-based learning algorithms for analog computing: a comparative study](https://mlsanity.com/paper/2312.15103/) (2023)
- [Brain-inspired Computational Intelligence via Predictive Coding](https://mlsanity.com/paper/2308.07870/) (2023)
- [Benchmarking Predictive Coding Networks -- Made Simple](https://mlsanity.com/paper/2407.01163/) (2024)
- [Training Deep Boltzmann Networks with Sparse Ising Machines](https://mlsanity.com/paper/2303.10728/) (2023)
- [A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks](https://mlsanity.com/paper/2212.00720/) (2022)
- [Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?](https://mlsanity.com/paper/2408.11979/) (2024)
- [Predictive Coding Networks and Inference Learning: Tutorial and Survey](https://mlsanity.com/paper/2407.04117/) (2024)
- [A Review of Neuroscience-Inspired Machine Learning](https://mlsanity.com/paper/2403.18929/) (2024)
- [Preventing Deterioration of Classification Accuracy in Predictive Coding Networks](https://mlsanity.com/paper/2208.07114/) (2022)
- [Active Sensing with Predictive Coding and Uncertainty Minimization](https://mlsanity.com/paper/2307.00668/) (2023)
- [Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation](https://mlsanity.com/paper/2411.02001/) (2024)
- [Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry](https://mlsanity.com/paper/2306.04810/) (2023)
- [Equilibrium Propagation for Non-Conservative Systems](https://mlsanity.com/paper/2602.03670/) (2026)
- [Training a Predictive Coding Network on ImageNet using Equilibrium Propagation](https://mlsanity.com/paper/2606.03584/) (2026)
- [Representation Learning in a Decomposed Encoder Design for Bio-inspired Hebbian Learning](https://mlsanity.com/paper/2401.08603/) (2023)
- [PhiNets: Brain-inspired Non-contrastive Learning Based on Temporal Prediction Hypothesis](https://mlsanity.com/paper/2405.14650/) (2024)
- [Self-orthogonalizing attractor neural networks emerging from the free energy principle](https://mlsanity.com/paper/2505.22749/) (2025)
- [Toward Practical Equilibrium Propagation: Brain-inspired Recurrent Neural Network with Feedback Regulation and Residual Connections](https://mlsanity.com/paper/2508.11659/) (2025)
- [On the Infinite Width and Depth Limits of Predictive Coding Networks](https://mlsanity.com/paper/2602.07697/) (2026)
- [Accelerated Predictive Coding Networks via Direct Kolen-Pollack Feedback Alignment](https://mlsanity.com/paper/2602.15571/) (2026)
- [Understanding Predictive Coding as an Adaptive Trust-Region Method](https://mlsanity.com/paper/2305.18188/) (2023)
- [Associative memory and dead neurons](https://mlsanity.com/paper/2410.13866/) (2024)
- [Biologically-Motivated Learning Model for Instructed Visual Processing](https://mlsanity.com/paper/2306.02415/) (2023)
- [Generalising E-prop to Deep Networks](https://mlsanity.com/paper/2512.24506/) (2025)
- [Equilibrium Propagation and Hamiltonian Inference in the Diffusive Fitzhugh-Nagumo Model](https://mlsanity.com/paper/2605.21568/) (2026)
- [Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Training](https://mlsanity.com/paper/2606.06539/) (2026)

## Abstract (arXiv)

How the brain performs credit assignment is a fundamental unsolved problem in neuroscience. Many `biologically plausible' algorithms have been proposed, which compute gradients that approximate those computed by backpropagation (BP), and which operate in ways that more closely satisfy the constraints imposed by neural circuitry. Many such algorithms utilize the framework of energy-based models (EBMs), in which all free variables in the model are optimized to minimize a global energy function. However, in the literature, these algorithms exist in isolation and no unified theory exists linking them together. Here, we provide a comprehensive theory of the conditions under which EBMs can approximate BP, which lets us unify many of the BP approximation results in the literature (namely, predictive coding, equilibrium propagation, and contrastive Hebbian learning) and demonstrate that their approximation to BP arises from a simple and general mathematical property of EBMs at free-phase equilibrium. This property can then be exploited in different ways with different energy functions, and these specific choices yield a family of BP-approximating algorithms, which both includes the known results in the literature and can be used to derive new ones.