mlsanityData to 7 Oct 2026

LocoProp: Enhancing BackProp via Local Loss Optimization

arXiv 2106.06199 ↗First posted 11 Jun 2021cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Second-order methods have shown state-of-the-art performance for optimizing deep neural networks. Nonetheless, their large memory requirement and high computational complexity, compared to first-order methods, hinder their versatility in a typical low-budget setup. This paper introduces a general framework of layerwise loss construction for multilayer neural networks that achieves a performance closer to second-order methods while utilizing first-order optimizers only. Our methodology lies upon a three-component loss, target, and regularizer combination, for which altering each component results in a new update rule. We provide examples using squared loss and layerwise Bregman divergences induced by the convex integral functions of various transfer functions. Our experiments on benchmark models and datasets validate the efficacy of our new approach, reducing the gap between first-order and second-order optimizers.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 23 of its 51 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

23 of 51 references.

References · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  4. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  5. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  6. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  7. 2014Difference Target Propagationcited by 144
  8. 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 135
  9. 2020Practical Quasi-Newton Methods for Training Deep Neural Networkscited by 67
  10. 2016Training Neural Networks Without Gradients: A Scalable ADMM Approachcited by 61
  11. 2019Robust Bi-Tempered Logistic Loss Based on Bregman Divergencescited by 61
  12. 2018Global Convergence of Block Coordinate Descent in Deep Learningcited by 38
11 more
  1. 2017Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networkscited by 26
  2. 2018Lifted Neural Networkscited by 19
  3. 2019Memory-Efficient Adaptive Optimizationcited by 19
  4. 2018Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Trainingcited by 18
  5. 2018A Proximal Block Coordinate Descent Algorithm for Deep Neural Network Trainingcited by 15
  6. 2018Lifted Proximal Operator Machinescited by 14
  7. 2019Contrastive Learning for Lifted Networkscited by 9
  8. 2017Proximal Backpropagationcited by 7
  9. 2020Guided Learning of Nonconvex Models through Successive Functional Gradient Optimizationcited by 7
  10. 2019An Implicit Form of Krasulina's k-PCA Update without the Orthonormality Constraintcited by 4
  11. 2016ParMAC: distributed optimisation of nested functions, with application to learning binary autoencoderscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 3; 2023: 5; 2024: 1; 2025: 2; 2026: 2.

15 papers.

Sort:

By area:

  1. 2021Large-Scale Differentially Private BERTcited by 117
  2. 2022Gradient Descent on Neurons and its Link to Approximate Second-Order Optimizationcited by 29
  3. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 14
  4. 2023Block-local learning with probabilistic latent representationscited by 6
  5. 2023Target-based Surrogates for Stochastic Optimizationcited by 4
  6. 2024Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagationcited by 4
  7. 2021Target Propagation via Regularized Inversioncited by 3
  8. 2022An alternative approach to train neural networks using monotone variational inequalitycited by 1
  9. 2022Layerwise Bregman Representation Learning with Applications to Knowledge Distillationcited by 1
  10. 2023FOSI: Hybrid First and Second Order Optimizationcited by 1
  11. 2023Data-efficient operator learning for solving high Mach number fluid flow problemscited by 0
  12. 2025Selective Matching Losses -- Not All Scores Are Created Equalcited by 0
3 more
  1. 2025An Adaptive Method Stabilizing Activations for Enhanced Generalizationcited by 0
  2. 2026Analog-Friendly Predictive Coding without Activation Derivativescited by 0
  3. 2026SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learningcited by 0