# Sequence Learning Using Equilibrium Propagation

arXiv: https://arxiv.org/abs/2209.09626 · HTML: https://mlsanity.com/paper/2209.09626/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 45 references that are in the map.
- First posted on arXiv 14 Sep 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks](https://mlsanity.com/paper/1701.05923/) (2017)
- [Dense Associative Memory for Pattern Recognition](https://mlsanity.com/paper/1606.01164/) (2016)
- [A Decomposable Attention Model for Natural Language Inference](https://mlsanity.com/paper/1606.01933/) (2016)
- [Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies](https://mlsanity.com/paper/2010.00951/) (2020)
- [Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias](https://mlsanity.com/paper/2006.03824/) (2020)
- [UnICORNN: A recurrent model for learning very long time dependencies](https://mlsanity.com/paper/2103.05487/) (2021)
- [Modern Hopfield Networks and Attention for Immune Repertoire Classification](https://mlsanity.com/paper/2007.13505/) (2020)
- [Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks](https://mlsanity.com/paper/1708.06834/) (2017)
- [Dense Associative Memory is Robust to Adversarial Inputs](https://mlsanity.com/paper/1701.00939/) (2017)
- [A Fast Unified Model for Parsing and Sentence Understanding](https://mlsanity.com/paper/1603.06021/) (2016)
- [Updates of Equilibrium Prop Match Gradients of Backprop Through Time in an RNN with Static Input](https://mlsanity.com/paper/1905.13633/) (2019)
- [Equivalence of Equilibrium Propagation and Recurrent Backpropagation](https://mlsanity.com/paper/1711.08416/) (2017)
- [EqSpike: Spike-driven Equilibrium Propagation for Neuromorphic Implementations](https://mlsanity.com/paper/2010.07859/) (2020)
- [Generalization of Equilibrium Propagation to Vector Field Dynamics](https://mlsanity.com/paper/1808.04873/) (2018)
- [Parallelizing Legendre Memory Unit Training](https://mlsanity.com/paper/2102.11417/) (2021)
- [DELTA: A DEep learning based Language Technology plAtform](https://mlsanity.com/paper/1908.01853/) (2019)

## Built on by (most cited first, 5 of 5)

- [Exploring Extreme Quantization in Spiking Language Models](https://mlsanity.com/paper/2405.02543/) (2024)
- [Deep Unsupervised Learning Using Spike-Timing-Dependent Plasticity](https://mlsanity.com/paper/2307.04054/) (2023)
- [Benchmarking Spiking Neural Network Learning Methods with Varying Locality](https://mlsanity.com/paper/2402.01782/) (2024)
- [Scaling Equilibrium Propagation to Deeper Neural Network Architectures](https://mlsanity.com/paper/2509.26003/) (2025)
- [Scalable Equilibrium Propagation via Intermediate Error Signals for Deep Convolutional CRNNs](https://mlsanity.com/paper/2508.15989/) (2025)

## Abstract (arXiv)

Equilibrium Propagation (EP) is a powerful and more bio-plausible alternative to conventional learning frameworks such as backpropagation. The effectiveness of EP stems from the fact that it relies only on local computations and requires solely one kind of computational unit during both of its training phases, thereby enabling greater applicability in domains such as bio-inspired neuromorphic computing. The dynamics of the model in EP is governed by an energy function and the internal states of the model consequently converge to a steady state following the state transition rules defined by the same. However, by definition, EP requires the input to the model (a convergent RNN) to be static in both the phases of training. Thus it is not possible to design a model for sequence classification using EP with an LSTM or GRU like architecture. In this paper, we leverage recent developments in modern hopfield networks to further understand energy based models and develop solutions for complex sequence classification tasks using EP while satisfying its convergence criteria and maintaining its theoretical similarities with recurrent backpropagation. We explore the possibility of integrating modern hopfield networks as an attention mechanism with convergent RNN models used in EP, thereby extending its applicability for the first time on two different sequence classification tasks in natural language processing viz. sentiment analysis (IMDB dataset) and natural language inference (SNLI dataset).