# Towards Scaling Difference Target Propagation by Learning Backprop Targets

arXiv: https://arxiv.org/abs/2201.13415 · HTML: https://mlsanity.com/paper/2201.13415/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2022.

## Built on by (most cited first, 30 of 34)

- [Holomorphic Equilibrium Propagation Computes Exact Gradients Through Finite Size Oscillations](https://mlsanity.com/paper/2209.00530/) (2022)
- [Brain-inspired Computational Intelligence via Predictive Coding](https://mlsanity.com/paper/2308.07870/) (2023)
- [Benchmarking Predictive Coding Networks -- Made Simple](https://mlsanity.com/paper/2407.01163/) (2024)
- [Training an Ising Machine with Equilibrium Propagation](https://mlsanity.com/paper/2305.18321/) (2023)
- [Blockwise Self-Supervised Learning at Scale](https://mlsanity.com/paper/2302.01647/) (2023)
- [A Theoretical Framework for Inference Learning](https://mlsanity.com/paper/2206.00164/) (2022)
- [Predictive Coding beyond Gaussian Distributions](https://mlsanity.com/paper/2211.03481/) (2022)
- [Dual Propagation: Accelerating Contrastive Hebbian Learning with Dyadic Neurons](https://mlsanity.com/paper/2302.01228/) (2023)
- [Improving equilibrium propagation without weight symmetry through Jacobian homeostasis](https://mlsanity.com/paper/2309.02214/) (2023)
- [Learning efficient backprojections across cortical hierarchies in real time](https://mlsanity.com/paper/2212.10249/) (2022)
- [Fixed-Weight Difference Target Propagation](https://mlsanity.com/paper/2212.10352/) (2022)
- [Block-local learning with probabilistic latent representations](https://mlsanity.com/paper/2305.14974/) (2023)
- [A Review of Neuroscience-Inspired Machine Learning](https://mlsanity.com/paper/2403.18929/) (2024)
- [Counter-Current Learning: A Biologically Plausible Dual Network Approach for Deep Learning](https://mlsanity.com/paper/2409.19841/) (2024)
- [Towards training digitally-tied analog blocks via hybrid gradient computation](https://mlsanity.com/paper/2409.03306/) (2024)
- [Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation](https://mlsanity.com/paper/2411.02001/) (2024)
- [Mono-Forward: Revisiting Forward-Forward through Objective-Locality Decomposition](https://mlsanity.com/paper/2501.09238/) (2025)
- [Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules](https://mlsanity.com/paper/2206.00823/) (2022)
- [Adaptive Spatial Goodness Encoding: Advancing and Scaling Forward-Forward Learning Without Backpropagation](https://mlsanity.com/paper/2509.12394/) (2025)
- [Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation](https://mlsanity.com/paper/2308.12053/) (2023)
- [Two Tales of Single-Phase Contrastive Hebbian Learning](https://mlsanity.com/paper/2402.08573/) (2024)
- [Covariance-Aware Goodness for Scalable Forward-Forward Learning](https://mlsanity.com/paper/2605.04346/) (2026)
- [HCL-FF: Hierarchical and Contrastive Learning for Forward-Forward Algorithm](https://mlsanity.com/paper/2605.24797/) (2026)
- [Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networks](https://mlsanity.com/paper/2306.01870/) (2023)
- [Spike-based alignment learning solves the weight transport problem](https://mlsanity.com/paper/2503.02642/) (2025)
- [Stochastic Layer-wise Learning: Scalable and Efficient Alternative to Backpropagation](https://mlsanity.com/paper/2505.05181/) (2025)
- [Learning without Global Backpropagation via Synergistic Information Distillation](https://mlsanity.com/paper/2510.03273/) (2025)
- [Biologically-Motivated Learning Model for Instructed Visual Processing](https://mlsanity.com/paper/2306.02415/) (2023)
- [Ubiquity of Emergent Hebbian Dynamics in Regularized Learning](https://mlsanity.com/paper/2505.18069/) (2025)
- [Forward Target Propagation: A Forward-Only Approach to Global Error Credit Assignment via Local Losses](https://mlsanity.com/paper/2506.11030/) (2025)

## Abstract (arXiv)

The development of biologically-plausible learning algorithms is important for understanding learning in the brain, but most of them fail to scale-up to real-world tasks, limiting their potential as explanations for learning by real brains. As such, it is important to explore learning algorithms that come with strong theoretical guarantees and can match the performance of backpropagation (BP) on complex tasks. One such algorithm is Difference Target Propagation (DTP), a biologically-plausible learning algorithm whose close relation with Gauss-Newton (GN) optimization has been recently established. However, the conditions under which this connection rigorously holds preclude layer-wise training of the feedback pathway synaptic weights (which is more biologically plausible). Moreover, good alignment between DTP weight updates and loss gradients is only loosely guaranteed and under very specific conditions for the architecture being trained. In this paper, we propose a novel feedback weight training scheme that ensures both that DTP approximates BP and that layer-wise feedback weight training can be restored without sacrificing any theoretical guarantees. Our theory is corroborated by experimental results and we report the best performance ever achieved by DTP on CIFAR-10 and ImageNet 32×32