# Lifted Bregman Training of Neural Networks

arXiv: https://arxiv.org/abs/2208.08772 · HTML: https://mlsanity.com/paper/2208.08772/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 79 references that are in the map.
- First posted on arXiv 18 Aug 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Training Neural Networks Without Gradients: A Scalable ADMM Approach](https://mlsanity.com/paper/1605.02026/) (2016)
- [A mathematical model for automatic differentiation in machine learning](https://mlsanity.com/paper/2006.02080/) (2020)
- [Deep Neural Network Structures Solving Variational Inequalities](https://mlsanity.com/paper/1808.07526/) (2018)
- [Lifted Neural Networks](https://mlsanity.com/paper/1805.01532/) (2018)
- [Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Training](https://mlsanity.com/paper/1811.08039/) (2018)
- [Convergence of Stochastic Proximal Gradient Algorithm](https://mlsanity.com/paper/1403.5074/) (2014)
- [A Bregman Learning Framework for Sparse Neural Networks](https://mlsanity.com/paper/2105.04319/) (2021)
- [Lifted Proximal Operator Machines](https://mlsanity.com/paper/1811.01501/) (2018)
- [Convex-Concave Backtracking for Inertial Bregman Proximal Gradient Algorithms in Non-Convex Optimization](https://mlsanity.com/paper/1904.03537/) (2019)
- [Contrastive Learning for Lifted Networks](https://mlsanity.com/paper/1905.02507/) (2019)
- [Choose your path wisely: gradient descent in a Bregman distance framework](https://mlsanity.com/paper/1712.04045/) (2017)
- [Parametric Majorization for Data-Driven Energy Minimization Methods](https://mlsanity.com/paper/1908.06209/) (2019)
- [Lifted Regression/Reconstruction Networks](https://mlsanity.com/paper/2005.03452/) (2020)
- [Generalised Perceptron Learning](https://mlsanity.com/paper/2012.03642/) (2020)
- [Bregman Distances in Inverse Problems and Partial Differential Equation](https://mlsanity.com/paper/1505.05191/) (2015)

## Built on by (most cited first, 10 of 10)

- [A primal-dual algorithm for image reconstruction with input-convex neural network regularizers](https://mlsanity.com/paper/2410.12441/) (2024)
- [Sparse Training of Neural Networks based on Multilevel Mirror Descent](https://mlsanity.com/paper/2602.03535/) (2026)
- [A Lifted Bregman Formulation for the Inversion of Deep Neural Networks](https://mlsanity.com/paper/2303.01965/) (2023)
- [On Model Compression for Neural Networks: Framework, Algorithm, and Convergence Guarantee](https://mlsanity.com/paper/2303.06815/) (2023)
- [Learning a Sparse Representation of Barron Functions with the Inverse Scale Space Flow](https://mlsanity.com/paper/2312.02671/) (2023)
- [HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation](https://mlsanity.com/paper/2603.14773/) (2026)
- [Augmented Lagrangian Predictive Coding](https://mlsanity.com/paper/2605.31022/) (2026)
- [A lifted Bregman strategy for training unfolded proximal neural network Gaussian denoisers](https://mlsanity.com/paper/2408.08742/) (2024)
- [Adaptive Regularization for Sparsity Control in Bregman-Based Optimizers](https://mlsanity.com/paper/2605.07892/) (2026)
- [Analog-Friendly Predictive Coding without Activation Derivatives](https://mlsanity.com/paper/2609.32350/) (2026)

## Abstract (arXiv)

We introduce a novel mathematical formulation for the training of feed-forward neural networks with (potentially non-smooth) proximal maps as activation functions. This formulation is based on Bregman distances and a key advantage is that its partial derivatives with respect to the network's parameters do not require the computation of derivatives of the network's activation functions. Instead of estimating the parameters with a combination of first-order optimisation method and back-propagation (as is the state-of-the-art), we propose the use of non-smooth first-order optimisation methods that exploit the specific structure of the novel formulation. We present several numerical results that demonstrate that these training approaches can be equally well or even better suited for the training of neural network-based classifiers and (denoising) autoencoders with sparse coding compared to more conventional training frameworks.