# Towards the Training of Deeper Predictive Coding Neural Networks

arXiv: https://arxiv.org/abs/2506.23800 · HTML: https://mlsanity.com/paper/2506.23800/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 51 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Dense Associative Memory for Pattern Recognition](https://mlsanity.com/paper/1606.01164/) (2016)
- [Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance](https://mlsanity.com/paper/2304.11127/) (2023)
- [Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](https://mlsanity.com/paper/2104.13386/) (2021)
- [How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation](https://mlsanity.com/paper/1407.7906/) (2014)
- [Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias](https://mlsanity.com/paper/2006.03824/) (2020)
- [Training End-to-End Analog Neural Networks with Equilibrium Propagation](https://mlsanity.com/paper/2006.01981/) (2020)
- [Training of Physical Neural Networks](https://mlsanity.com/paper/2406.03372/) (2024)
- [The Neural Coding Framework for Learning Generative Models](https://mlsanity.com/paper/2012.03405/) (2020)
- [Associative Memories via Predictive Coding](https://mlsanity.com/paper/2109.08063/) (2021)
- [Holomorphic Equilibrium Propagation Computes Exact Gradients Through Finite Size Oscillations](https://mlsanity.com/paper/2209.00530/) (2022)
- [Energy-based learning algorithms for analog computing: a comparative study](https://mlsanity.com/paper/2312.15103/) (2023)
- [Brain-inspired Computational Intelligence via Predictive Coding](https://mlsanity.com/paper/2308.07870/) (2023)
- [Benchmarking Predictive Coding Networks -- Made Simple](https://mlsanity.com/paper/2407.01163/) (2024)
- [Equilibrium Propagation with Continual Weight Updates](https://mlsanity.com/paper/2005.04168/) (2020)
- [Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation](https://mlsanity.com/paper/2205.03076/) (2022)
- [Reverse Differentiation via Predictive Coding](https://mlsanity.com/paper/2103.04689/) (2021)
- [Predictive Coding beyond Gaussian Distributions](https://mlsanity.com/paper/2211.03481/) (2022)
- [Agnostic Physics-Driven Deep Learning](https://mlsanity.com/paper/2205.15021/) (2022)
- [μPC: Scaling Predictive Coding to 100+ Layer Networks](https://mlsanity.com/paper/2505.13124/) (2025)
- [Quantum Equilibrium Propagation: Gradient-Descent Training of Quantum Systems](https://mlsanity.com/paper/2406.00879/) (2024)
- [Towards training digitally-tied analog blocks via hybrid gradient computation](https://mlsanity.com/paper/2409.03306/) (2024)
- [Divide-and-Conquer Predictive Coding: a structured Bayesian inference algorithm](https://mlsanity.com/paper/2408.05834/) (2024)
- [Predictive coding, precision and natural gradients](https://mlsanity.com/paper/2111.06942/) (2021)

## Built on by (most cited first, 7 of 7)

- [Faster Predictive Coding Networks via Better Initialization](https://mlsanity.com/paper/2601.20895/) (2026)
- [Training a Predictive Coding Network on ImageNet using Equilibrium Propagation](https://mlsanity.com/paper/2606.03584/) (2026)
- [On the Infinite Width and Depth Limits of Predictive Coding Networks](https://mlsanity.com/paper/2602.07697/) (2026)
- [Closed-form predictive coding via hierarchical Gaussian filters](https://mlsanity.com/paper/2605.20293/) (2026)
- [Hebbian Learning with Global Direction](https://mlsanity.com/paper/2601.21367/) (2026)
- [Error Highways: Scaling Predictive Coding to Very Deep Networks](https://mlsanity.com/paper/2606.22744/) (2026)
- [Analog-Friendly Predictive Coding without Activation Derivatives](https://mlsanity.com/paper/2609.32350/) (2026)

## Abstract (arXiv)

Predictive coding networks are neural models that perform inference through an iterative energy minimization process, whose operations are local in space and time. While effective in shallow architectures, they suffer significant performance degradation beyond five to seven layers. In this work, we show that this degradation is caused by exponentially imbalanced errors between layers during weight updates, and by predictions from the previous layers not being effective in guiding updates in deeper layers. Furthermore, when training models with skip connections, the energy propagated by the residuals reaches higher layers faster than that propagated by the main pathway, affecting test accuracy. We address the first issue by introducing a novel precision-weighted optimization of latent variables that balances error distributions during the relaxation phase, the second issue by proposing a novel weight update mechanism that reduces error accumulation in deeper layers, and the third one by using auxiliary neurons that slow down the propagation of the energy in the residual connections. Empirically, our methods achieve performance comparable to backpropagation on deep models such as ResNets, opening new possibilities for predictive coding in complex tasks.