mlsanityData to 13 Oct 2026

Reverse Differentiation via Predictive Coding

arXiv 2103.04689 ↗First posted 8 Mar 2021cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

Deep learning has redefined the field of artificial intelligence (AI) thanks to the rise of artificial neural networks, which are architectures inspired by their neurological counterpart in the brain. Through the years, this dualism between AI and neuroscience has brought immense benefits to both fields, allowing neural networks to be used in dozens of applications. These networks use an efficient implementation of reverse differentiation, called backpropagation (BP). This algorithm, however, is often criticized for its biological implausibility (e.g., lack of local update rules for the parameters). Therefore, biologically plausible learning methods that rely on predictive coding (PC), a framework for describing information processing in the brain, are increasingly studied. Recent works prove that these methods can approximate BP up to a certain margin on multilayer perceptrons (MLPs), and asymptotically on any other complex model, and that zero-divergence inference learning (Z-IL), a variant of PC, is able to exactly implement BP on MLPs. However, the recent literature shows also that there is no biologically plausible method yet that can exactly replicate the weight update of BP on complex models. To fill this gap, in this paper, we generalize (PC and) Z-IL by directly defining them on computational graphs, and show that it can perform exact reverse differentiation. What results is the first biologically plausible algorithm that is equivalent to BP in the way of updating parameters on any neural network, providing a bridge between the interdisciplinary research of neuroscience and deep learning.

Key facts. Cited by 23 papers in the map (data through 13 Oct 2026). Builds on 20 of its 49 references that are in the map. First posted on arXiv 8 Mar 2021.

Builds on

20 of 49 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  4. 2016Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagationcited by 255
  5. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  6. 2014Difference Target Propagationcited by 146
  7. 2019Training Neural Networks with Local Error Signalscited by 135
  8. 2018Dendritic cortical microcircuits approximate the backpropagation algorithmcited by 97
  9. 2018The Singular Values of Convolutional Layerscited by 95
  10. 2020Predictive Coding Approximates Backprop along Arbitrary Computation Graphscited by 89
  11. 2014How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagationcited by 79
  12. 2018Biologically-plausible learning algorithms can scale to large datasetscited by 37
8 more
  1. 2019Biologically plausible deep learning -- but how far can we go with shallow networks?cited by 25
  2. 2018Generalization of Equilibrium Propagation to Vector Field Dynamicscited by 22
  3. 2018Deep learning with asymmetric connections and Hebbian updatescited by 16
  4. 2017Learning to Adapt by Minimizing Discrepancycited by 12
  5. 2015A Theory of Local Learning, the Learning Channel, and the Optimality of Backpropagationcited by 11
  6. 2019Structured and Deep Similarity Matching via Structured and Deep Hebbian Networkscited by 9
  7. 2019Cortical credit assignment by Hebbian, neuromodulatory and inhibitory plasticitycited by 8
  8. 2018Dictionary Learning by Dynamical Neural Networkscited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 7; 2023: 4; 2024: 3; 2025: 4; 2026: 5.

23 papers.

Sort:

By area:

  1. 2023The Predictive Forward-Forward Algorithmcited by 35
  2. 2022Learning on Arbitrary Graph Topologies via Predictive Codingcited by 33
  3. 2023Brain-inspired Computational Intelligence via Predictive Codingcited by 30
  4. 2022Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learningcited by 26
  5. 2022A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networkscited by 21
  6. 2022A Theoretical Framework for Inference and Learning in Predictive Coding Networkscited by 17
  7. 2022Predictive Coding beyond Gaussian Distributionscited by 16
  8. 2022Convolutional Neural Generative Coding: Scaling Predictive Coding to Natural Imagescited by 10
  9. 2024Predictive Coding Networks and Inference Learning: Tutorial and Surveycited by 7
  10. 2025Towards the Training of Deeper Predictive Coding Neural Networkscited by 7
  11. 2023Predictive Coding beyond Correlationscited by 6
  12. 2024A Review of Neuroscience-Inspired Machine Learningcited by 6
11 more
  1. 2022Robust Graph Representation Learning via Predictive Codingcited by 5
  2. 2024Tight Stability, Convergence, and Robustness Bounds for Predictive Coding Networkscited by 3
  3. 2026Faster Predictive Coding Networks via Better Initializationcited by 3
  4. 2025Bayesian Predictive Codingcited by 2
  5. 2025Bridging Predictive Coding and MDL: A Two-Part Code Framework for Deep Learningcited by 1
  6. 2026Predictive Coding Graphs are a Superset of Feedforward Neural Networkscited by 1
  7. 2026Backpropagation-Free Trunk Training via the Split Forward Gradientscited by 1
  8. 2023Biologically-Motivated Learning Model for Instructed Visual Processingcited by 0
  9. 2025DMPCN: Dynamic Modulated Predictive Coding Network with Hybrid Feedback Representationscited by 0
  10. 2026Learning Long-Range Dependencies with Temporal Predictive Codingcited by 0
  11. 2026Learning in Deep Networks under Dale's Constraintcited by 0