# Learning with Local Gradients at the Edge

arXiv: https://arxiv.org/abs/2208.08503 · HTML: https://mlsanity.com/paper/2208.08503/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 17 Aug 2022.

## Builds on (most cited first, 16 of 16)

- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Deep Learning in Spiking Neural Networks](https://mlsanity.com/paper/1804.08150/) (2018)
- [Neural Architecture Search without Training](https://mlsanity.com/paper/2006.04647/) (2020)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [Professor Forcing: A New Algorithm for Training Recurrent Networks](https://mlsanity.com/paper/1610.09038/) (2016)
- [Advances in Pre-Training Distributed Word Representations](https://mlsanity.com/paper/1712.09405/) (2017)
- [A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning](https://mlsanity.com/paper/2006.06224/) (2020)
- [Training Neural Networks Without Gradients: A Scalable ADMM Approach](https://mlsanity.com/paper/1605.02026/) (2016)
- [Deep supervised learning using local errors](https://mlsanity.com/paper/1711.06756/) (2017)
- [ADMM for Efficient Deep Learning with Global Convergence](https://mlsanity.com/paper/1905.13611/) (2019)
- [Hyperdimensional Feature Fusion for Out-Of-Distribution Detection](https://mlsanity.com/paper/2112.05341/) (2021)
- [A 28-nm Convolutional Neuromorphic Processor Enabling Online Learning with Spike-Based Retinas](https://mlsanity.com/paper/2005.06318/) (2020)
- [Toward Model Parallelism for Deep Neural Network based on Gradient-free ADMM Framework](https://mlsanity.com/paper/2009.02868/) (2020)
- [ZORB: A Derivative-Free Backpropagation Algorithm for Neural Networks](https://mlsanity.com/paper/2011.08895/) (2020)
- [BitNet: Bit-Regularized Deep Neural Networks](https://mlsanity.com/paper/1708.04788/) (2017)

## Built on by (most cited first, 1 of 1)

- [Block-local learning with probabilistic latent representations](https://mlsanity.com/paper/2305.14974/) (2023)

## Abstract (arXiv)

To enable learning on edge devices with fast convergence and low memory, we present a novel backpropagation-free optimization algorithm dubbed Target Projection Stochastic Gradient Descent (tpSGD). tpSGD generalizes direct random target projection to work with arbitrary loss functions and extends target projection for training recurrent neural networks (RNNs) in addition to feedforward networks. tpSGD uses layer-wise stochastic gradient descent (SGD) and local targets generated via random projections of the labels to train the network layer-by-layer with only forward passes. tpSGD doesn't require retaining gradients during optimization, greatly reducing memory allocation compared to SGD backpropagation (BP) methods that require multiple instances of the entire neural network weights, input/output, and intermediate results. Our method performs comparably to BP gradient-descent within 5% accuracy on relatively shallow networks of fully connected layers, convolutional layers, and recurrent layers. tpSGD also outperforms other state-of-the-art gradient-free algorithms in shallow models consisting of multi-layer perceptrons, convolutional neural networks (CNNs), and RNNs with competitive accuracy and less memory and time. We evaluate the performance of tpSGD in training deep neural networks (e.g. VGG) and extend the approach to multi-layer RNNs. These experiments highlight new research directions related to optimized layer-based adaptor training for domain-shift using tpSGD at the edge.