# Early Inference in Energy-Based Models Approximates Back-Propagation

arXiv: https://arxiv.org/abs/1510.02777 · HTML: https://mlsanity.com/paper/1510.02777/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2015.

## Built on by (most cited first, 13 of 13)

- [Predictive Coding: a Theoretical and Experimental Review](https://mlsanity.com/paper/2107.12979/) (2021)
- [Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learning](https://mlsanity.com/paper/2206.02629/) (2022)
- [A Theoretical Framework for Inference and Learning in Predictive Coding Networks](https://mlsanity.com/paper/2207.12316/) (2022)
- [Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignment](https://mlsanity.com/paper/2312.09257/) (2023)
- [Applications of the Free Energy Principle to Machine Learning and Neuroscience](https://mlsanity.com/paper/2107.00140/) (2021)
- [Learning efficient backprojections across cortical hierarchies in real time](https://mlsanity.com/paper/2212.10249/) (2022)
- [Unlocking Deep Learning: A BP-Free Approach for Parallel Block-Wise Training of Neural Networks](https://mlsanity.com/paper/2312.13311/) (2023)
- [Accelerating Hopfield Network Dynamics: Beyond Synchronous Updates and Forward Euler](https://mlsanity.com/paper/2311.15673/) (2023)
- [RAL2M: Retrieval Augmented Learning-To-Match Against Hallucination in Compliance-Guaranteed Service Systems](https://mlsanity.com/paper/2601.02917/) (2026)
- [Training a Hopfield Variational Autoencoder with Equilibrium Propagation](https://mlsanity.com/paper/2311.15047/) (2023)
- [aSTDP: A More Biologically Plausible Learning](https://mlsanity.com/paper/2206.14137/) (2022)
- [Does Feedback Alignment Work at Biological Timescales?](https://mlsanity.com/paper/2510.18808/) (2025)
- [Equilibrium Propagation and Hamiltonian Inference in the Diffusive Fitzhugh-Nagumo Model](https://mlsanity.com/paper/2605.21568/) (2026)

## Abstract (arXiv)

We show that Langevin MCMC inference in an energy-based model with latent variables has the property that the early steps of inference, starting from a stationary point, correspond to propagating error gradients into internal layers, similarly to back-propagation. The error that is back-propagated is with respect to visible units that have received an outside driving force pushing them away from the stationary point. Back-propagated error gradients correspond to temporal derivatives of the activation of hidden units. This observation could be an element of a theory for explaining how brains perform credit assignment in deep hierarchies as efficiently as back-propagation does. In this theory, the continuous-valued latent variables correspond to averaged voltage potential (across time, spikes, and possibly neurons in the same minicolumn), and neural computation corresponds to approximate inference and error back-propagation at the same time.