# A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks

arXiv: https://arxiv.org/abs/2212.00720 · HTML: https://mlsanity.com/paper/2212.00720/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 75 references that are in the map.
- First posted on arXiv 16 Nov 2022.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One](https://mlsanity.com/paper/1912.03263/) (2019)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Dendritic cortical microcircuits approximate the backpropagation algorithm](https://mlsanity.com/paper/1810.11393/) (2018)
- [Predictive Coding Approximates Backprop along Arbitrary Computation Graphs](https://mlsanity.com/paper/2006.04182/) (2020)
- [Predictive Coding: a Theoretical and Experimental Review](https://mlsanity.com/paper/2107.12979/) (2021)
- [Posterior calibration and exploratory analysis for natural language processing models](https://mlsanity.com/paper/1508.05154/) (2015)
- [The free energy principle for action and perception: A mathematical review](https://mlsanity.com/paper/1705.09156/) (2017)
- [Training End-to-End Analog Neural Networks with Equilibrium Propagation](https://mlsanity.com/paper/2006.01981/) (2020)
- [Hebbian Deep Learning Without Feedback](https://mlsanity.com/paper/2209.11883/) (2022)
- [Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?](https://mlsanity.com/paper/2202.09467/) (2022)
- [The Neural Coding Framework for Learning Generative Models](https://mlsanity.com/paper/2012.03405/) (2020)
- [Associative Memories via Predictive Coding](https://mlsanity.com/paper/2109.08063/) (2021)
- [Biologically Motivated Algorithms for Propagating Local Target Representations](https://mlsanity.com/paper/1805.11703/) (2018)
- [Biologically-plausible learning algorithms can scale to large datasets](https://mlsanity.com/paper/1811.03567/) (2018)
- [Learning on Arbitrary Graph Topologies via Predictive Coding](https://mlsanity.com/paper/2201.13180/) (2022)
- [Continual Learning of Recurrent Neural Networks by Locally Aligning Distributed Representations](https://mlsanity.com/paper/1810.07411/) (2018)
- [Equilibrium Propagation with Continual Weight Updates](https://mlsanity.com/paper/2005.04168/) (2020)
- [Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learning](https://mlsanity.com/paper/2206.02629/) (2022)
- [Reverse Differentiation via Predictive Coding](https://mlsanity.com/paper/2103.04689/) (2021)
- [Predify: Augmenting deep neural networks with brain-inspired predictive coding dynamics](https://mlsanity.com/paper/2106.02749/) (2021)
- [A Theoretical Framework for Inference Learning](https://mlsanity.com/paper/2206.00164/) (2022)

## Built on by (most cited first, 22 of 22)

- [Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignment](https://mlsanity.com/paper/2312.09257/) (2023)
- [Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?](https://mlsanity.com/paper/2408.11979/) (2024)
- [Predictive Coding Networks and Inference Learning: Tutorial and Survey](https://mlsanity.com/paper/2407.04117/) (2024)
- [Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation](https://mlsanity.com/paper/2411.02001/) (2024)
- [Divide-and-Conquer Predictive Coding: a structured Bayesian inference algorithm](https://mlsanity.com/paper/2408.05834/) (2024)
- [Introduction to Predictive Coding Networks for Machine Learning](https://mlsanity.com/paper/2506.06332/) (2025)
- [On the Infinite Width and Depth Limits of Predictive Coding Networks](https://mlsanity.com/paper/2602.07697/) (2026)
- [Accelerated Predictive Coding Networks via Direct Kolen-Pollack Feedback Alignment](https://mlsanity.com/paper/2602.15571/) (2026)
- [Predictive Coding Graphs are a Superset of Feedforward Neural Networks](https://mlsanity.com/paper/2603.06142/) (2026)
- [Self-supervised local learning rules learn the hidden hierarchical structure of high-dimensional data](https://mlsanity.com/paper/2605.18557/) (2026)
- [Closed-form predictive coding via hierarchical Gaussian filters](https://mlsanity.com/paper/2605.20293/) (2026)
- [DMPCN: Dynamic Modulated Predictive Coding Network with Hybrid Feedback Representations](https://mlsanity.com/paper/2504.14665/) (2025)
- [Semantic and episodic memories in a predictive coding model of the neocortex](https://mlsanity.com/paper/2509.01987/) (2025)
- [Can Local Learning Match Self-Supervised Backpropagation?](https://mlsanity.com/paper/2601.21683/) (2026)
- [Learning Long-Range Dependencies with Temporal Predictive Coding](https://mlsanity.com/paper/2602.18131/) (2026)
- [A Synthesizable RTL Implementation of Predictive Coding Networks](https://mlsanity.com/paper/2603.18066/) (2026)
- [Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Training](https://mlsanity.com/paper/2606.06539/) (2026)
- [Error Highways: Scaling Predictive Coding to Very Deep Networks](https://mlsanity.com/paper/2606.22744/) (2026)
- [Weight Feedback Computes the Jacobian Transpose Locally in Modern Deep Networks](https://mlsanity.com/paper/2607.13380/) (2026)
- [Single State Update Predictive Coding training for Time Series Forecasting and Anomaly Detection](https://mlsanity.com/paper/2608.24697/) (2026)
- [A Study of Hidden-State Optimization Order in Predictive Coding Networks](https://mlsanity.com/paper/2609.00686/) (2026)
- [DeltaNet-PC: Test-Time Memory Without Backpropagation Through Time](https://mlsanity.com/paper/2610.15185/) (2026)

## Abstract (arXiv)

Predictive coding networks are neuroscience-inspired models with roots in both Bayesian statistics and neuroscience. Training such models, however, is quite inefficient and unstable. In this work, we show how by simply changing the temporal scheduling of the update rule for the synaptic weights leads to an algorithm that is much more efficient and stable than the original one, and has theoretical guarantees in terms of convergence. The proposed algorithm, that we call incremental predictive coding (iPC) is also more biologically plausible than the original one, as it it fully automatic. In an extensive set of experiments, we show that iPC constantly performs better than the original formulation on a large number of benchmarks for image classification, as well as for the training of both conditional and masked language models, in terms of test accuracy, efficiency, and convergence with respect to a large set of hyperparameters.