mlsanityData to 7 Oct 2026

Difference Target Propagation

arXiv 1412.7525 ↗First posted 23 Dec 2014cs.LG cs.NEcited by 144

Not on the map. Why?

Abstract (arXiv)

Back-propagation has been the workhorse of recent successes of deep learning but it relies on infinitesimal effects (partial derivatives) in order to perform credit assignment. This could become a serious issue as one considers deeper and more non-linear functions, e.g., consider the extreme case of nonlinearity where the relation between parameters and cost is actually discrete. Inspired by the biological implausibility of back-propagation, a few approaches have been proposed in the past that could play a similar credit assignment role. In this spirit, we explore a novel approach to credit assignment in deep networks that we call target propagation. The main idea is to compute targets rather than gradients, at each layer. Like gradients, they are propagated backwards. In a way that is related but different from previously proposed proxies for back-propagation which rely on a backwards network with symmetric weights, target propagation relies on auto-encoders at each layer. Unlike back-propagation, it can be applied even when units exchange stochastic bits rather than real numbers. We show that a linear correction for the imperfectness of the auto-encoders, called difference target propagation, is very effective to make target propagation actually work, leading to results comparable to back-propagation for deep networks with discrete and continuous units and denoising auto-encoders and achieving state of the art for stochastic networks.

Key facts. Cited by 144 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Dec 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 23; 2023: 21; 2024: 21; 2025: 22; 2026: 30.

144 papers.

Sort:

By area:

  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  2. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  3. 2021Predictive Coding: a Theoretical and Experimental Reviewcited by 74
  4. 2021On Training Implicit Modelscited by 73
  5. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  6. 2022Scaling Forward Gradient With Local Lossescited by 65
  7. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  8. 2022Holomorphic Equilibrium Propagation Computes Exact Gradients Through Finite Size Oscillationscited by 40
  9. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  10. 2023The Predictive Forward-Forward Algorithmcited by 35
  11. 2023Brain-inspired Computational Intelligence via Predictive Codingcited by 30
  12. 2021Credit Assignment in Neural Networks through Deep Feedback Controlcited by 28
38 more
  1. 2022Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learningcited by 26
  2. 2023Online learning of long-range dependenciescited by 25
  3. 2022Signal Propagation: A Framework for Learning and Inference In a Forward Passcited by 24
  4. 2021Reverse Differentiation via Predictive Codingcited by 23
  5. 2023Blockwise Self-Supervised Learning at Scalecited by 22
  6. 2021Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligencecited by 21
  7. 2021The Backpropagation Algorithm Implemented on Spiking Neuromorphic Hardwarecited by 19
  8. 2022Single-phase deep learning in cortico-cortical networkscited by 19
  9. 2022A Theoretical Framework for Inference Learningcited by 17
  10. 2022The least-control principle for local learning at equilibriumcited by 16
  11. 2022A Theoretical Framework for Inference and Learning in Predictive Coding Networkscited by 16
  12. 2022Predictive Coding beyond Gaussian Distributionscited by 16
  13. 2021LocoProp: Enhancing BackProp via Local Loss Optimizationcited by 15
  14. 2021Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networkscited by 14
  15. 2023A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Servicescited by 14
  16. 2022Deep Incubation: Training Large Models by Divide-and-Conqueringcited by 13
  17. 2023Forward Learning with Top-Down Feedback: Empirical and Analytical Characterizationcited by 13
  18. 2021Backprop-Free Reinforcement Learning with Active Neural Generative Codingcited by 12
  19. 2023Dual Propagation: Accelerating Contrastive Hebbian Learning with Dyadic Neuronscited by 12
  20. 2021Predictive coding feedback results in perceived illusory contours in a recurrent neural networkcited by 10
  21. 2021Meta-Learning Bidirectional Update Rulescited by 10
  22. 2022Minimizing Control for Credit Assignment with Strong Feedbackcited by 10
  23. 2022Convolutional Neural Generative Coding: Scaling Predictive Coding to Natural Imagescited by 10
  24. 2021Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithmscited by 9
  25. 2022Is Bio-Inspired Learning Better than Backprop? Benchmarking Bio Learning vs. Backpropcited by 9
  26. 2024Momentum Auxiliary Network for Supervised Local Learningcited by 9
  27. 2022Learning efficient backprojections across cortical hierarchies in real timecited by 8
  28. 2023A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functionscited by 8
  29. 2023Improving equilibrium propagation without weight symmetry through Jacobian homeostasiscited by 8
  30. 2024End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Trainingcited by 8
  31. 2024Scaling Supervised Local Learning with Augmented Auxiliary Networkscited by 8
  32. 2025NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagationcited by 8
  33. 2024Predictive Coding Networks and Inference Learning: Tutorial and Surveycited by 7
  34. 2021A Gradient Estimator for Time-Varying Electrical Networks with Non-Linear Dissipationcited by 6
  35. 2021Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networkscited by 6
  36. 2022Active Predicting Coding: Brain-Inspired Reinforcement Learning for Sparse Reward Robotic Control Problemscited by 6
  37. 2022Fixed-Weight Difference Target Propagationcited by 6
  38. 2023Block-local learning with probabilistic latent representationscited by 6

50 most cited of 144 All 144 (JSON)