# LocoProp: Enhancing BackProp via Local Loss Optimization

arXiv: https://arxiv.org/abs/2106.06199 · HTML: https://mlsanity.com/paper/2106.06199/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 51 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Shampoo: Preconditioned Stochastic Tensor Optimization](https://mlsanity.com/paper/1802.09568/) (2018)
- [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/1905.12558/) (2019)
- [Difference Target Propagation](https://mlsanity.com/paper/1412.7525/) (2014)
- [Decoupled Neural Interfaces using Synthetic Gradients](https://mlsanity.com/paper/1608.05343/) (2016)
- [Practical Quasi-Newton Methods for Training Deep Neural Networks](https://mlsanity.com/paper/2006.08877/) (2020)
- [Training Neural Networks Without Gradients: A Scalable ADMM Approach](https://mlsanity.com/paper/1605.02026/) (2016)
- [Robust Bi-Tempered Logistic Loss Based on Bregman Divergences](https://mlsanity.com/paper/1906.03361/) (2019)
- [Global Convergence of Block Coordinate Descent in Deep Learning](https://mlsanity.com/paper/1803.00225/) (2018)
- [Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks](https://mlsanity.com/paper/1711.07354/) (2017)
- [Lifted Neural Networks](https://mlsanity.com/paper/1805.01532/) (2018)
- [Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Training](https://mlsanity.com/paper/1811.08039/) (2018)
- [Memory-Efficient Adaptive Optimization](https://mlsanity.com/paper/1901.11150/) (2019)
- [A Proximal Block Coordinate Descent Algorithm for Deep Neural Network Training](https://mlsanity.com/paper/1803.09082/) (2018)
- [Lifted Proximal Operator Machines](https://mlsanity.com/paper/1811.01501/) (2018)
- [Contrastive Learning for Lifted Networks](https://mlsanity.com/paper/1905.02507/) (2019)
- [Proximal Backpropagation](https://mlsanity.com/paper/1706.04638/) (2017)
- [Guided Learning of Nonconvex Models through Successive Functional Gradient Optimization](https://mlsanity.com/paper/2006.16840/) (2020)
- [An Implicit Form of Krasulina's k-PCA Update without the Orthonormality Constraint](https://mlsanity.com/paper/1909.04803/) (2019)
- [ParMAC: distributed optimisation of nested functions, with application to learning binary autoencoders](https://mlsanity.com/paper/1605.09114/) (2016)

## Built on by (most cited first, 15 of 15)

- [Large-Scale Differentially Private BERT](https://mlsanity.com/paper/2108.01624/) (2021)
- [Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization](https://mlsanity.com/paper/2201.12250/) (2022)
- [On the Parameterization of Second-Order Optimization Effective Towards the Infinite Width](https://mlsanity.com/paper/2312.12226/) (2023)
- [Block-local learning with probabilistic latent representations](https://mlsanity.com/paper/2305.14974/) (2023)
- [Target-based Surrogates for Stochastic Optimization](https://mlsanity.com/paper/2302.02607/) (2023)
- [Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation](https://mlsanity.com/paper/2411.02001/) (2024)
- [Target Propagation via Regularized Inversion](https://mlsanity.com/paper/2112.01453/) (2021)
- [An alternative approach to train neural networks using monotone variational inequality](https://mlsanity.com/paper/2202.08876/) (2022)
- [Layerwise Bregman Representation Learning with Applications to Knowledge Distillation](https://mlsanity.com/paper/2209.07080/) (2022)
- [FOSI: Hybrid First and Second Order Optimization](https://mlsanity.com/paper/2302.08484/) (2023)
- [Data-efficient operator learning for solving high Mach number fluid flow problems](https://mlsanity.com/paper/2311.16860/) (2023)
- [Selective Matching Losses -- Not All Scores Are Created Equal](https://mlsanity.com/paper/2506.04446/) (2025)
- [An Adaptive Method Stabilizing Activations for Enhanced Generalization](https://mlsanity.com/paper/2506.08353/) (2025)
- [Analog-Friendly Predictive Coding without Activation Derivatives](https://mlsanity.com/paper/2609.32350/) (2026)
- [SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learning](https://mlsanity.com/paper/2609.35440/) (2026)

## Abstract (arXiv)

Second-order methods have shown state-of-the-art performance for optimizing deep neural networks. Nonetheless, their large memory requirement and high computational complexity, compared to first-order methods, hinder their versatility in a typical low-budget setup. This paper introduces a general framework of layerwise loss construction for multilayer neural networks that achieves a performance closer to second-order methods while utilizing first-order optimizers only. Our methodology lies upon a three-component loss, target, and regularizer combination, for which altering each component results in a new update rule. We provide examples using squared loss and layerwise Bregman divergences induced by the convex integral functions of various transfer functions. Our experiments on benchmark models and datasets validate the efficacy of our new approach, reducing the gap between first-order and second-order optimizers.