# Lifted Proximal Operator Machines

arXiv: https://arxiv.org/abs/1811.01501 · HTML: https://mlsanity.com/paper/1811.01501/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2018.

## Built on by (most cited first, 14 of 14)

- [Offsite-Tuning: Transfer Learning without Full Model](https://mlsanity.com/paper/2302.04870/) (2023)
- [On Training Implicit Models](https://mlsanity.com/paper/2111.05177/) (2021)
- [RU-Net: Regularized Unrolling Network for Scene Graph Generation](https://mlsanity.com/paper/2205.01297/) (2022)
- [LocoProp: Enhancing BackProp via Local Loss Optimization](https://mlsanity.com/paper/2106.06199/) (2021)
- [Dual Propagation: Accelerating Contrastive Hebbian Learning with Dyadic Neurons](https://mlsanity.com/paper/2302.01228/) (2023)
- [Deep Incubation: Training Large Models by Divide-and-Conquering](https://mlsanity.com/paper/2212.04129/) (2022)
- [Lifted Bregman Training of Neural Networks](https://mlsanity.com/paper/2208.08772/) (2022)
- [Monotone, Bi-Lipschitz, and Polyak-Lojasiewicz Networks](https://mlsanity.com/paper/2402.01344/) (2024)
- [Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methods](https://mlsanity.com/paper/2101.03419/) (2021)
- [Bilevel Programs Meet Deep Learning: A Unifying View on Inference Learning Methods](https://mlsanity.com/paper/2105.07231/) (2021)
- [Inertial Proximal Deep Learning Alternating Minimization for Efficient Neutral Network Training](https://mlsanity.com/paper/2102.00267/) (2021)
- [A primal-dual algorithm for image reconstruction with input-convex neural network regularizers](https://mlsanity.com/paper/2410.12441/) (2024)
- [Augmented Lagrangian Predictive Coding](https://mlsanity.com/paper/2605.31022/) (2026)
- [Strengthening the Internal Adversarial Robustness in Lifted Neural Networks](https://mlsanity.com/paper/2503.07818/) (2025)

## Abstract (arXiv)

We propose a new optimization method for training feed-forward neural networks. By rewriting the activation function as an equivalent proximal operator, we approximate a feed-forward neural network by adding the proximal operators to the objective function as penalties, hence we call the lifted proximal operator machine (LPOM). LPOM is block multi-convex in all layer-wise weights and activations. This allows us to use block coordinate descent to update the layer-wise weights and activations in parallel. Most notably, we only use the mapping of the activation function itself, rather than its derivatives, thus avoiding the gradient vanishing or blow-up issues in gradient based training methods. So our method is applicable to various non-decreasing Lipschitz continuous activation functions, which can be saturating and non-differentiable. LPOM does not require more auxiliary variables than the layer-wise activations, thus using roughly the same amount of memory as stochastic gradient descent (SGD) does. We further prove the convergence of updating the layer-wise weights and activations. Experiments on MNIST and CIFAR-10 datasets testify to the advantages of LPOM.