# Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks

arXiv: https://arxiv.org/abs/2103.03725 · HTML: https://mlsanity.com/paper/2103.03725/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 35 references that are in the map.
- First posted on arXiv 5 Mar 2021.

## Builds on (most cited first, 11 of 11)

- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [Difference Target Propagation](https://mlsanity.com/paper/1412.7525/) (2014)
- [Training Neural Networks with Local Error Signals](https://mlsanity.com/paper/1901.06656/) (2019)
- [The Singular Values of Convolutional Layers](https://mlsanity.com/paper/1805.10408/) (2018)
- [Predictive Coding Approximates Backprop along Arbitrary Computation Graphs](https://mlsanity.com/paper/2006.04182/) (2020)
- [Biologically Motivated Algorithms for Propagating Local Target Representations](https://mlsanity.com/paper/1805.11703/) (2018)
- [Biologically plausible deep learning -- but how far can we go with shallow networks?](https://mlsanity.com/paper/1905.04101/) (2019)
- [Learning to Adapt by Minimizing Discrepancy](https://mlsanity.com/paper/1711.11542/) (2017)
- [A Theory of Local Learning, the Learning Channel, and the Optimality of Backpropagation](https://mlsanity.com/paper/1506.06472/) (2015)
- [Cortical credit assignment by Hebbian, neuromodulatory and inhibitory plasticity](https://mlsanity.com/paper/1911.00307/) (2019)

## Built on by (most cited first, 14 of 14)

- [Predictive Coding: a Theoretical and Experimental Review](https://mlsanity.com/paper/2107.12979/) (2021)
- [Associative Memories via Predictive Coding](https://mlsanity.com/paper/2109.08063/) (2021)
- [On the relationship between predictive coding and backpropagation](https://mlsanity.com/paper/2106.13082/) (2021)
- [A Theoretical Framework for Inference and Learning in Predictive Coding Networks](https://mlsanity.com/paper/2207.12316/) (2022)
- [Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization](https://mlsanity.com/paper/2302.05440/) (2023)
- [Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?](https://mlsanity.com/paper/2408.11979/) (2024)
- [Predictive Coding as a Neuromorphic Alternative to Backpropagation: A Critical Evaluation](https://mlsanity.com/paper/2304.02658/) (2023)
- [Predictive Coding Networks and Inference Learning: Tutorial and Survey](https://mlsanity.com/paper/2407.04117/) (2024)
- [Active Sensing with Predictive Coding and Uncertainty Minimization](https://mlsanity.com/paper/2307.00668/) (2023)
- [Meta predictive learning model of languages in neural circuits](https://mlsanity.com/paper/2309.04106/) (2023)
- [On the Infinite Width and Depth Limits of Predictive Coding Networks](https://mlsanity.com/paper/2602.07697/) (2026)
- [Understanding Predictive Coding as an Adaptive Trust-Region Method](https://mlsanity.com/paper/2305.18188/) (2023)
- [Feature Alignment as a Generative Process](https://mlsanity.com/paper/2106.12562/) (2021)
- [Deep hybrid models: infer and plan in a dynamic world](https://mlsanity.com/paper/2402.10088/) (2024)

## Abstract (arXiv)

Predictive coding networks (PCNs) are an influential model for information processing in the brain. They have appealing theoretical interpretations and offer a single mechanism that accounts for diverse perceptual phenomena of the brain. On the other hand, backpropagation (BP) is commonly regarded to be the most successful learning method in modern machine learning. Thus, it is exciting that recent work formulates inference learning (IL) that trains PCNs to approximate BP. However, there are several remaining critical issues: (i) IL is an approximation to BP with unrealistic/non-trivial requirements, (ii) IL approximates BP in single-step weight updates; whether it leads to the same point as BP after the weight updates are conducted for more steps is unknown, and (iii) IL is computationally significantly more costly than BP. To solve these issues, a variant of IL that is strictly equivalent to BP in fully connected networks has been proposed. In this work, we build on this result by showing that it also holds for more complex architectures, namely, convolutional neural networks and (many-to-one) recurrent neural networks. To our knowledge, we are the first to show that a biologically plausible algorithm is able to exactly replicate the accuracy of BP on such complex architectures, bridging the existing gap between IL and BP, and setting an unprecedented performance for PCNs, which can now be considered as efficient alternatives to BP.