# Lifted Neural Networks

arXiv: https://arxiv.org/abs/1805.01532 · HTML: https://mlsanity.com/paper/1805.01532/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 May 2018.

## Built on by (most cited first, 20 of 20)

- [Offsite-Tuning: Transfer Learning without Full Model](https://mlsanity.com/paper/2302.04870/) (2023)
- [Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape](https://mlsanity.com/paper/2201.08025/) (2022)
- [LocoProp: Enhancing BackProp via Local Loss Optimization](https://mlsanity.com/paper/2106.06199/) (2021)
- [Dual Propagation: Accelerating Contrastive Hebbian Learning with Dyadic Neurons](https://mlsanity.com/paper/2302.01228/) (2023)
- [Deep Incubation: Training Large Models by Divide-and-Conquering](https://mlsanity.com/paper/2212.04129/) (2022)
- [Lifted Bregman Training of Neural Networks](https://mlsanity.com/paper/2208.08772/) (2022)
- [Spiking Neural Networks and Bio-Inspired Supervised Deep Learning: A Survey](https://mlsanity.com/paper/2307.16235/) (2023)
- [Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methods](https://mlsanity.com/paper/2101.03419/) (2021)
- [Bilevel Programs Meet Deep Learning: A Unifying View on Inference Learning Methods](https://mlsanity.com/paper/2105.07231/) (2021)
- [Inertial Proximal Deep Learning Alternating Minimization for Efficient Neutral Network Training](https://mlsanity.com/paper/2102.00267/) (2021)
- [A Mixed-Integer Programming Approach to Training Dense Neural Networks](https://mlsanity.com/paper/2201.00723/) (2022)
- [A primal-dual algorithm for image reconstruction with input-convex neural network regularizers](https://mlsanity.com/paper/2410.12441/) (2024)
- [Stochastic Block-ADMM for Training Deep Networks](https://mlsanity.com/paper/2105.00339/) (2021)
- [Personalized On-Device E-health Analytics with Decentralized Block Coordinate Descent](https://mlsanity.com/paper/2112.09341/) (2021)
- [Two Tales of Single-Phase Contrastive Hebbian Learning](https://mlsanity.com/paper/2402.08573/) (2024)
- [Implicit Bias in Matrix Factorization and its Explicit Realization in a New Architecture](https://mlsanity.com/paper/2501.16322/) (2025)
- [Augmented Lagrangian Predictive Coding](https://mlsanity.com/paper/2605.31022/) (2026)
- [Block Coordinate Descent for Neural Networks Provably Finds Global Minima](https://mlsanity.com/paper/2510.22667/) (2025)
- [ProtoSeam: Lifting Classifier Training with Latent Gaussian Mixture Models](https://mlsanity.com/paper/2609.35174/) (2026)
- [Taming Nonconvexity with Epigraph Lifting](https://mlsanity.com/paper/2610.16429/) (2026)

## Abstract (arXiv)

We describe a novel family of models of multi- layer feedforward neural networks in which the activation functions are encoded via penalties in the training problem. Our approach is based on representing a non-decreasing activation function as the argmin of an appropriate convex optimiza- tion problem. The new framework allows for algo- rithms such as block-coordinate descent methods to be applied, in which each step is composed of a simple (no hidden layer) supervised learning problem that is parallelizable across data points and/or layers. Experiments indicate that the pro- posed models provide excellent initial guesses for weights for standard neural networks. In addi- tion, the model provides avenues for interesting extensions, such as robustness against noisy in- puts and optimizing over parameters in activation functions.