# Local Pairwise Distance Matching for Backpropagation-Free Reinforcement Learning

arXiv: https://arxiv.org/abs/2507.11367 · HTML: https://mlsanity.com/paper/2507.11367/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 33 of its 63 references that are in the map.
- First posted on arXiv 15 Jul 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 33)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Difference Target Propagation](https://mlsanity.com/paper/1412.7525/) (2014)
- [Decoupled Neural Interfaces using Synthetic Gradients](https://mlsanity.com/paper/1608.05343/) (2016)
- [Training Neural Networks with Local Error Signals](https://mlsanity.com/paper/1901.06656/) (2019)
- [Representation Matters: Offline Pretraining for Sequential Decision Making](https://mlsanity.com/paper/2102.05815/) (2021)
- [How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation](https://mlsanity.com/paper/1407.7906/) (2014)
- [Gradients without Backpropagation](https://mlsanity.com/paper/2202.08587/) (2022)
- [Scaling Forward Gradient With Local Losses](https://mlsanity.com/paper/2210.03310/) (2022)
- [Continual Learning: Applications and the Road Forward](https://mlsanity.com/paper/2311.11908/) (2023)
- [A Theoretical Framework for Target Propagation](https://mlsanity.com/paper/2006.14331/) (2020)
- [Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation?](https://mlsanity.com/paper/2202.09467/) (2022)
- [The Neural Coding Framework for Learning Generative Models](https://mlsanity.com/paper/2012.03405/) (2020)
- [Biologically Motivated Algorithms for Propagating Local Target Representations](https://mlsanity.com/paper/1805.11703/) (2018)
- [Unsupervised Representation Learning by Predicting Random Distances](https://mlsanity.com/paper/1912.12186/) (2019)
- [The Predictive Forward-Forward Algorithm](https://mlsanity.com/paper/2301.01452/) (2023)
- [The Role of Baselines in Policy Gradient Optimization](https://mlsanity.com/paper/2301.06276/) (2023)
- [Leveraging the two timescale regime to demonstrate convergence of neural networks](https://mlsanity.com/paper/2304.09576/) (2023)
- [Unsupervised Representation Learning in Deep Reinforcement Learning: A Review](https://mlsanity.com/paper/2208.14226/) (2022)
- [Pretraining in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2211.03959/) (2022)
- [Error Forward-Propagation: Reusing Feedforward Connections to Propagate Errors in Deep Learning](https://mlsanity.com/paper/1808.03357/) (2018)
- [Layer-wise training of deep networks using kernel similarity](https://mlsanity.com/paper/1703.07115/) (2017)
- [Backprop-Free Reinforcement Learning with Active Neural Generative Coding](https://mlsanity.com/paper/2107.07046/) (2021)
- [Evolutionary Training and Abstraction Yields Algorithmic Generalization of Neural Computers](https://mlsanity.com/paper/2105.07957/) (2021)
- [Hierarchical Predictive Coding Models in a Deep-Learning Framework](https://mlsanity.com/paper/2005.03230/) (2020)

## Abstract (arXiv)

Training neural networks with reinforcement learning (RL) typically relies on backpropagation (BP), necessitating storage of activations from the forward pass for subsequent backward updates. Furthermore, backpropagating error signals through multiple layers often leads to vanishing or exploding gradients, which can degrade learning performance and stability. We propose a novel approach that trains each layer of the neural network using local signals during the forward pass in RL settings. Our approach introduces local, layer-wise losses leveraging the principle of matching pairwise distances from multi-dimensional scaling, enhanced with optional reward-driven guidance. This method allows each hidden layer to be trained using local signals computed during forward propagation, thus eliminating the need for backward passes and storing intermediate activations. Our experiments, conducted with policy gradient methods across common RL benchmarks, demonstrate that this backpropagation-free method achieves competitive performance compared to their classical BP-based counterpart. Additionally, the proposed method enhances stability and consistency within and across runs, and improves performance especially in challenging environments.