mlsanityData to 7 Oct 2026

A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks

arXiv 2212.00720 ↗First posted 16 Nov 2022cs.NE cs.AI cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Predictive coding networks are neuroscience-inspired models with roots in both Bayesian statistics and neuroscience. Training such models, however, is quite inefficient and unstable. In this work, we show how by simply changing the temporal scheduling of the update rule for the synaptic weights leads to an algorithm that is much more efficient and stable than the original one, and has theoretical guarantees in terms of convergence. The proposed algorithm, that we call incremental predictive coding (iPC) is also more biologically plausible than the original one, as it it fully automatic. In an extensive set of experiments, we show that iPC constantly performs better than the original formulation on a large number of benchmarks for image classification, as well as for the training of both conditional and masked language models, in terms of test accuracy, efficiency, and convergence with respect to a large set of hyperparameters.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 36 of its 75 references that are in the map. First posted on arXiv 16 Nov 2022.

Builds on

36 of 75 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  8. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  9. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  10. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  11. 2016Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagationcited by 251
  12. 2018Dendritic cortical microcircuits approximate the backpropagation algorithmcited by 96
24 more
  1. 2020Predictive Coding Approximates Backprop along Arbitrary Computation Graphscited by 89
  2. 2015Posterior calibration and exploratory analysis for natural language processing modelscited by 75
  3. 2021Predictive Coding: a Theoretical and Experimental Reviewcited by 74
  4. 2017The free energy principle for action and perception: A mathematical reviewcited by 66
  5. 2022Hebbian Deep Learning Without Feedbackcited by 61
  6. 2020Training End-to-End Analog Neural Networks with Equilibrium Propagationcited by 57
  7. 2022Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?cited by 45
  8. 2020The Neural Coding Framework for Learning Generative Modelscited by 44
  9. 2018Biologically Motivated Algorithms for Propagating Local Target Representationscited by 43
  10. 2021Associative Memories via Predictive Codingcited by 42
  11. 2018Biologically-plausible learning algorithms can scale to large datasetscited by 37
  12. 2022Learning on Arbitrary Graph Topologies via Predictive Codingcited by 33
  13. 2018Continual Learning of Recurrent Neural Networks by Locally Aligning Distributed Representationscited by 32
  14. 2022Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learningcited by 26
  15. 2020Equilibrium Propagation with Continual Weight Updatescited by 25
  16. 2021Reverse Differentiation via Predictive Codingcited by 23
  17. 2021Predify: Augmenting deep neural networks with brain-inspired predictive coding dynamicscited by 19
  18. 2022A Theoretical Framework for Inference Learningcited by 17
  19. 2022A Theoretical Framework for Inference and Learning in Predictive Coding Networkscited by 16
  20. 2022Predictive Coding beyond Gaussian Distributionscited by 16
  21. 2019Lifelong Neural Predictive Coding: Learning Cumulatively Online without Forgettingcited by 15
  22. 2017Learning to Adapt by Minimizing Discrepancycited by 12
  23. 2019On the Global Convergence of (Fast) Incremental Expectation Maximization Methodscited by 12
  24. 2020Relaxing the Constraints on Predictive Coding Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 3; 2026: 13.

21 papers.

Sort:

  1. 2023Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignmentcited by 13
  2. 2024Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?cited by 10
  3. 2024Predictive Coding Networks and Inference Learning: Tutorial and Surveycited by 7
  4. 2024Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagationcited by 4
  5. 2024Divide-and-Conquer Predictive Coding: a structured Bayesian inference algorithmcited by 3
  6. 2025Introduction to Predictive Coding Networks for Machine Learningcited by 2
  7. 2026On the Infinite Width and Depth Limits of Predictive Coding Networkscited by 2
  8. 2026Accelerated Predictive Coding Networks via Direct Kolen-Pollack Feedback Alignmentcited by 2
  9. 2026Predictive Coding Graphs are a Superset of Feedforward Neural Networkscited by 1
  10. 2026Self-supervised local learning rules learn the hidden hierarchical structure of high-dimensional datacited by 1
  11. 2026Closed-form predictive coding via hierarchical Gaussian filterscited by 1
  12. 2025DMPCN: Dynamic Modulated Predictive Coding Network with Hybrid Feedback Representationscited by 0
9 more
  1. 2025Semantic and episodic memories in a predictive coding model of the neocortexcited by 0
  2. 2026Can Local Learning Match Self-Supervised Backpropagation?cited by 0
  3. 2026Learning Long-Range Dependencies with Temporal Predictive Codingcited by 0
  4. 2026A Synthesizable RTL Implementation of Predictive Coding Networkscited by 0
  5. 2026Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Trainingcited by 0
  6. 2026Error Highways: Scaling Predictive Coding to Very Deep Networkscited by 0
  7. 2026Weight Feedback Computes the Jacobian Transpose Locally in Modern Deep Networkscited by 0
  8. 2026Single State Update Predictive Coding training for Time Series Forecasting and Anomaly Detectioncited by 0
  9. 2026A Study of Hidden-State Optimization Order in Predictive Coding Networkscited by 0