# Deriving Differential Target Propagation from Iterating Approximate Inverses

arXiv: https://arxiv.org/abs/2007.15139 · HTML: https://mlsanity.com/paper/2007.15139/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jul 2020.

## Built on by (most cited first, 15 of 15)

- [Gradients without Backpropagation](https://mlsanity.com/paper/2202.08587/) (2022)
- [Holomorphic Equilibrium Propagation Computes Exact Gradients Through Finite Size Oscillations](https://mlsanity.com/paper/2209.00530/) (2022)
- [Credit Assignment in Neural Networks through Deep Feedback Control](https://mlsanity.com/paper/2106.07887/) (2021)
- [A Theoretical Framework for Inference Learning](https://mlsanity.com/paper/2206.00164/) (2022)
- [A Theoretical Framework for Inference and Learning in Predictive Coding Networks](https://mlsanity.com/paper/2207.12316/) (2022)
- [Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignment](https://mlsanity.com/paper/2312.09257/) (2023)
- [Minimizing Control for Credit Assignment with Strong Feedback](https://mlsanity.com/paper/2204.07249/) (2022)
- [Applications of the Free Energy Principle to Machine Learning and Neuroscience](https://mlsanity.com/paper/2107.00140/) (2021)
- [Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithms](https://mlsanity.com/paper/2108.13446/) (2021)
- [Fixed-Weight Difference Target Propagation](https://mlsanity.com/paper/2212.10352/) (2022)
- [A Review of Neuroscience-Inspired Machine Learning](https://mlsanity.com/paper/2403.18929/) (2024)
- [Gradient-Free Training of Recurrent Neural Networks using Random Perturbations](https://mlsanity.com/paper/2405.08967/) (2024)
- [Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methods](https://mlsanity.com/paper/2101.03419/) (2021)
- [Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation](https://mlsanity.com/paper/2411.02001/) (2024)
- [Target Propagation via Regularized Inversion](https://mlsanity.com/paper/2112.01453/) (2021)

## Abstract (arXiv)

We show that a particular form of target propagation, i.e., relying on learned inverses of each layer, which is differential, i.e., where the target is a small perturbation of the forward propagation, gives rise to an update rule which corresponds to an approximate Gauss-Newton gradient-based optimization, without requiring the manipulation or inversion of large matrices. What is interesting is that this is more biologically plausible than back-propagation yet may turn out to implicitly provide a stronger optimization procedure. Extending difference target propagation, we consider several iterative calculations based on local auto-encoders at each layer in order to achieve more precise inversions for more accurate target propagation and we show that these iterative procedures converge exponentially fast if the auto-encoding function minus the identity function has a Lipschitz constant smaller than one, i.e., the auto-encoder is coarsely succeeding at performing an inversion. We also propose a way to normalize the changes at each layer to take into account the relative influence of each layer on the output, so that larger weight changes are done on more influential layers, like would happen in ordinary back-propagation with gradient descent.