# A deep learning theory for neural networks grounded in physics

arXiv: https://arxiv.org/abs/2103.09985 · HTML: https://mlsanity.com/paper/2103.09985/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 126 references that are in the map.
- First posted on arXiv 18 Mar 2021.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [signSGD: Compressed Optimisation for Non-Convex Problems](https://mlsanity.com/paper/1802.04434/) (2018)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Meta-Learning: A Survey](https://mlsanity.com/paper/1810.03548/) (2018)
- [SuperSpike: Supervised learning in multi-layer spiking neural networks](https://mlsanity.com/paper/1705.11146/) (2017)
- [Training End-to-End Analog Neural Networks with Equilibrium Propagation](https://mlsanity.com/paper/2006.01981/) (2020)
- [Theory of Deep Learning III: explaining the non-overfitting puzzle](https://mlsanity.com/paper/1801.00173/) (2017)
- [EqSpike: Spike-driven Equilibrium Propagation for Neuromorphic Implementations](https://mlsanity.com/paper/2010.07859/) (2020)
- [Equilibrium Propagation with Continual Weight Updates](https://mlsanity.com/paper/2005.04168/) (2020)
- [Generalization of Equilibrium Propagation to Vector Field Dynamics](https://mlsanity.com/paper/1808.04873/) (2018)
- [Feedforward Initialization for Fast Inference of Deep Generative Networks is biologically plausible](https://mlsanity.com/paper/1606.01651/) (2016)
- [A Gradient Estimator for Time-Varying Electrical Networks with Non-Linear Dissipation](https://mlsanity.com/paper/2103.05636/) (2021)
- [Cognitive Action Laws: The Case of Visual Features](https://mlsanity.com/paper/1808.09162/) (2018)
- [Mitigating Asymmetric Nonlinear Weight Update Effects in Hardware Neural Network based on Analog Resistive Synapse](https://mlsanity.com/paper/1712.05895/) (2017)
- [Equilibrium Propagation for Complete Directed Neural Networks](https://mlsanity.com/paper/2006.08798/) (2020)

## Built on by (most cited first, 23 of 23)

- [Energy-based learning algorithms for analog computing: a comparative study](https://mlsanity.com/paper/2312.15103/) (2023)
- [Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation](https://mlsanity.com/paper/2205.03076/) (2022)
- [The least-control principle for local learning at equilibrium](https://mlsanity.com/paper/2207.01332/) (2022)
- [Agnostic Physics-Driven Deep Learning](https://mlsanity.com/paper/2205.15021/) (2022)
- [A contrastive rule for meta-learning](https://mlsanity.com/paper/2104.01677/) (2021)
- [Dual Propagation: Accelerating Contrastive Hebbian Learning with Dyadic Neurons](https://mlsanity.com/paper/2302.01228/) (2023)
- [Learning by non-interfering feedback chemical signaling in physical networks](https://mlsanity.com/paper/2203.12098/) (2022)
- [Frequency propagation: Multi-mechanism learning in nonlinear physical networks](https://mlsanity.com/paper/2208.08862/) (2022)
- [Bounds all around: training energy-based models with bidirectional bounds](https://mlsanity.com/paper/2111.00929/) (2021)
- [Quantum Equilibrium Propagation: Gradient-Descent Training of Quantum Systems](https://mlsanity.com/paper/2406.00879/) (2024)
- [Quantum Equilibrium Propagation for efficient training of quantum systems based on Onsager reciprocity](https://mlsanity.com/paper/2406.06482/) (2024)
- [Equilibrium Propagation for Learning in Lagrangian Dynamical Systems](https://mlsanity.com/paper/2505.07363/) (2025)
- [Towards training digitally-tied analog blocks via hybrid gradient computation](https://mlsanity.com/paper/2409.03306/) (2024)
- [A First-order Generative Bilevel Optimization Framework for Diffusion Models](https://mlsanity.com/paper/2502.08808/) (2025)
- [Training a Predictive Coding Network on ImageNet using Equilibrium Propagation](https://mlsanity.com/paper/2606.03584/) (2026)
- [Convergence of energy-based learning in linear resistive networks](https://mlsanity.com/paper/2503.00349/) (2025)
- [KirchhoffNet: A Scalable Ultra Fast Analog Neural Network](https://mlsanity.com/paper/2310.15872/) (2023)
- [Harnessing intuitive local evolution rules for physical learning](https://mlsanity.com/paper/2507.19561/) (2025)
- [Asymptotically Stable Quaternion-valued Hopfield-structured Neural Network with Periodic Projection-based Supervised Learning Rules](https://mlsanity.com/paper/2510.16607/) (2025)
- [Emergent learning in physical systems as feedback-based aging in a glassy landscape](https://mlsanity.com/paper/2309.04382/) (2023)
- [Nature-Inspired Local Propagation](https://mlsanity.com/paper/2402.05959/) (2024)
- [Exploring bidirectional bounds for minimax-training of Energy-based models](https://mlsanity.com/paper/2506.04609/) (2025)
- [Perturbative Contrastive Physical Learning](https://mlsanity.com/paper/2606.09756/) (2026)

## Abstract (arXiv)

In the last decade, deep learning has become a major component of artificial intelligence. The workhorse of deep learning is the optimization of loss functions by stochastic gradient descent (SGD). Traditionally in deep learning, neural networks are differentiable mathematical functions, and the loss gradients required for SGD are computed with the backpropagation algorithm. However, the computer architectures on which these neural networks are implemented and trained suffer from speed and energy inefficiency issues, due to the separation of memory and processing in these architectures. To solve these problems, the field of neuromorphic computing aims at implementing neural networks on hardware architectures that merge memory and processing, just like brains do. In this thesis, we argue that building large, fast and efficient neural networks on neuromorphic architectures also requires rethinking the algorithms to implement and train them. We present an alternative mathematical framework, also compatible with SGD, which offers the possibility to design neural networks in substrates that directly exploit the laws of physics. Our framework applies to a very broad class of models, namely those whose state or dynamics are described by variational equations. This includes physical systems whose equilibrium state minimizes an energy function, and physical systems whose trajectory minimizes an action functional. We present a simple procedure to compute the loss gradients in such systems, called equilibrium propagation (EqProp), which requires solely locally available information for each trainable parameter. Since many models in physics and engineering can be described by variational principles, our framework has the potential to be applied to a broad variety of physical systems whose applications extend to various fields of engineering, beyond neuromorphic computing.