# Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?

arXiv: https://arxiv.org/abs/2408.11979 · HTML: https://mlsanity.com/paper/2408.11979/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 65 references that are in the map.
- First posted on arXiv 21 Aug 2024.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [How to Escape Saddle Points Efficiently](https://mlsanity.com/paper/1703.00887/) (2017)
- [Deep Learning without Poor Local Minima](https://mlsanity.com/paper/1605.07110/) (2016)
- [Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsity](https://mlsanity.com/paper/2106.15933/) (2021)
- [Predictive Coding: a Theoretical and Experimental Review](https://mlsanity.com/paper/2107.12979/) (2021)
- [Optimization for deep learning: theory and algorithms](https://mlsanity.com/paper/1912.08957/) (2019)
- [A Theoretical Framework for Target Propagation](https://mlsanity.com/paper/2006.14331/) (2020)
- [Analytic Insights into Structure and Rank of Neural Network Hessian Maps](https://mlsanity.com/paper/2106.16225/) (2021)
- [Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?](https://mlsanity.com/paper/2202.09467/) (2022)
- [Energy-based learning algorithms for analog computing: a comparative study](https://mlsanity.com/paper/2312.15103/) (2023)
- [Escaping Saddle Points with Adaptive Gradient Methods](https://mlsanity.com/paper/1901.09149/) (2019)
- [Global optimality conditions for deep neural networks](https://mlsanity.com/paper/1707.02444/) (2017)
- [Brain-inspired Computational Intelligence via Predictive Coding](https://mlsanity.com/paper/2308.07870/) (2023)
- [Benchmarking Predictive Coding Networks -- Made Simple](https://mlsanity.com/paper/2407.01163/) (2024)
- [Depth Creates No Bad Local Minima](https://mlsanity.com/paper/1702.08580/) (2017)
- [Hybrid Predictive Coding: Inferring, Fast and Slow](https://mlsanity.com/paper/2204.02169/) (2022)
- [Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learning](https://mlsanity.com/paper/2206.02629/) (2022)
- [Scaling Down Deep Learning with MNIST-1D](https://mlsanity.com/paper/2011.14439/) (2020)
- [The loss landscape of deep linear neural networks: a second-order analysis](https://mlsanity.com/paper/2107.13289/) (2021)
- [Exact Solutions of a Deep Linear Network](https://mlsanity.com/paper/2202.04777/) (2022)
- [A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks](https://mlsanity.com/paper/2212.00720/) (2022)
- [A Theoretical Framework for Inference Learning](https://mlsanity.com/paper/2206.00164/) (2022)
- [A Theoretical Framework for Inference and Learning in Predictive Coding Networks](https://mlsanity.com/paper/2207.12316/) (2022)
- [Predictive Coding beyond Gaussian Distributions](https://mlsanity.com/paper/2211.03481/) (2022)
- [Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks](https://mlsanity.com/paper/2103.03725/) (2021)
- [Understanding and Improving Optimization in Predictive Coding Networks](https://mlsanity.com/paper/2305.13562/) (2023)
- [On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points](https://mlsanity.com/paper/1902.04811/) (2019)
- [Predictive Coding beyond Correlations](https://mlsanity.com/paper/2306.15479/) (2023)

## Built on by (most cited first, 10 of 10)

- [Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architectures](https://mlsanity.com/paper/2512.20607/) (2025)
- [ePC: Fast and Deep Predictive Coding in Digital Simulation](https://mlsanity.com/paper/2505.20137/) (2025)
- [Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation](https://mlsanity.com/paper/2411.02001/) (2024)
- [JPC: Flexible Inference for Predictive Coding Networks in JAX](https://mlsanity.com/paper/2412.03676/) (2024)
- [Bayesian Predictive Coding](https://mlsanity.com/paper/2503.24016/) (2025)
- [On the Infinite Width and Depth Limits of Predictive Coding Networks](https://mlsanity.com/paper/2602.07697/) (2026)
- [Augmented Lagrangian Predictive Coding](https://mlsanity.com/paper/2605.31022/) (2026)
- [Learning Long-Range Dependencies with Temporal Predictive Coding](https://mlsanity.com/paper/2602.18131/) (2026)
- [Understanding Sample Efficiency in Predictive Coding](https://mlsanity.com/paper/2605.11911/) (2026)
- [Analog-Friendly Predictive Coding without Activation Derivatives](https://mlsanity.com/paper/2609.32350/) (2026)

## Abstract (arXiv)

Predictive coding (PC) is an energy-based learning algorithm that performs iterative inference over network activities before updating weights. Recent work suggests that PC can converge in fewer learning steps than backpropagation thanks to its inference procedure. However, these advantages are not always observed, and the impact of PC inference on learning is not theoretically well understood. Here, we study the geometry of the PC energy landscape at the inference equilibrium of the network activities. For deep linear networks, we first show that the equilibrated energy is simply a rescaled mean squared error loss with a weight-dependent rescaling. We then prove that many highly degenerate (non-strict) saddles of the loss including the origin become much easier to escape (strict) in the equilibrated energy. Our theory is validated by experiments on both linear and non-linear networks. Based on these and other results, we conjecture that all the saddles of the equilibrated energy are strict. Overall, this work suggests that PC inference makes the loss landscape more benign and robust to vanishing gradients, while also highlighting the fundamental challenge of scaling PC to deeper models.