# Beyond Backprop: Online Alternating Minimization with Auxiliary Variables

arXiv: https://arxiv.org/abs/1806.09077 · HTML: https://mlsanity.com/paper/1806.09077/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jun 2018.

## Built on by (most cited first, 23 of 23)

- [Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications](https://mlsanity.com/paper/2111.12444/) (2021)
- [Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape](https://mlsanity.com/paper/2201.08025/) (2022)
- [A Theoretical Framework for Inference Learning](https://mlsanity.com/paper/2206.00164/) (2022)
- [Latent Equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons](https://mlsanity.com/paper/2110.14549/) (2021)
- [Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methods](https://mlsanity.com/paper/2101.03419/) (2021)
- [Finding High-Value Training Data Subset through Differentiable Convex Programming](https://mlsanity.com/paper/2104.13794/) (2021)
- [Gradient Forward-Propagation for Large-Scale Temporal Video Modelling](https://mlsanity.com/paper/2106.08318/) (2021)
- [Convergence Rates of Training Deep Neural Networks via Alternating Minimization Methods](https://mlsanity.com/paper/2208.14318/) (2022)
- [Local Learning with Neuron Groups](https://mlsanity.com/paper/2301.07635/) (2023)
- [Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagation](https://mlsanity.com/paper/2411.02001/) (2024)
- [Stochastic Block-ADMM for Training Deep Networks](https://mlsanity.com/paper/2105.00339/) (2021)
- [Layer-Parallel Training of Residual Networks with Auxiliary-Variable Networks](https://mlsanity.com/paper/2112.05387/) (2021)
- [Duality Principle and Biologically Plausible Learning: Connecting the Representer Theorem and Hebbian Learning](https://mlsanity.com/paper/2309.16687/) (2023)
- [Two Tales of Single-Phase Contrastive Hebbian Learning](https://mlsanity.com/paper/2402.08573/) (2024)
- [A Convergent ADMM Framework for Efficient Neural Network Training](https://mlsanity.com/paper/2112.11619/) (2021)
- [Deep Regression Ensembles](https://mlsanity.com/paper/2203.05417/) (2022)
- [A Triple-Inertial Accelerated Alternating Optimization Method for Deep Learning Training](https://mlsanity.com/paper/2503.08489/) (2025)
- [Etat de l'art sur l'application des bandits multi-bras](https://mlsanity.com/paper/2101.00001/) (2021)
- [Learning DNN networks using un-rectifying ReLU with compressed sensing application](https://mlsanity.com/paper/2101.06940/) (2021)
- [Front Contribution instead of Back Propagation](https://mlsanity.com/paper/2106.05569/) (2021)
- [ADMM Algorithms for Residual Network Training: Convergence Analysis and Parallel Implementation](https://mlsanity.com/paper/2310.15334/) (2023)
- [Strengthening the Internal Adversarial Robustness in Lifted Neural Networks](https://mlsanity.com/paper/2503.07818/) (2025)
- [Neural Network Training via Stochastic Alternating Minimization with Trainable Step Sizes](https://mlsanity.com/paper/2508.04193/) (2025)

## Abstract (arXiv)

Despite significant recent advances in deep neural networks, training them remains a challenge due to the highly non-convex nature of the objective function. State-of-the-art methods rely on error backpropagation, which suffers from several well-known issues, such as vanishing and exploding gradients, inability to handle non-differentiable nonlinearities and to parallelize weight-updates across layers, and biological implausibility. These limitations continue to motivate exploration of alternative training algorithms, including several recently proposed auxiliary-variable methods which break the complex nested objective function into local subproblems. However, those techniques are mainly offline (batch), which limits their applicability to extremely large datasets, as well as to online, continual or reinforcement learning. The main contribution of our work is a novel online (stochastic/mini-batch) alternating minimization (AM) approach for training deep neural networks, together with the first theoretical convergence guarantees for AM in stochastic settings and promising empirical results on a variety of architectures and datasets.