mlsanityData to 14 Oct 2026

Lifted Bregman Training of Neural Networks

arXiv 2208.08772 ↗First posted 18 Aug 2022math.OC cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

We introduce a novel mathematical formulation for the training of feed-forward neural networks with (potentially non-smooth) proximal maps as activation functions. This formulation is based on Bregman distances and a key advantage is that its partial derivatives with respect to the network's parameters do not require the computation of derivatives of the network's activation functions. Instead of estimating the parameters with a combination of first-order optimisation method and back-propagation (as is the state-of-the-art), we propose the use of non-smooth first-order optimisation methods that exploit the specific structure of the novel formulation. We present several numerical results that demonstrate that these training approaches can be equally well or even better suited for the training of neural network-based classifiers and (denoising) autoencoders with sparse coding compared to more conventional training frameworks.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 20 of its 79 references that are in the map. First posted on arXiv 18 Aug 2022.

Builds on

20 of 79 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016Layer Normalizationcited by 5,669
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  6. 2016Training Neural Networks Without Gradients: A Scalable ADMM Approachcited by 61
  7. 2020A mathematical model for automatic differentiation in machine learningcited by 45
  8. 2018Deep Neural Network Structures Solving Variational Inequalitiescited by 26
  9. 2018Lifted Neural Networkscited by 20
  10. 2018Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Trainingcited by 19
  11. 2014Convergence of Stochastic Proximal Gradient Algorithmcited by 17
  12. 2021A Bregman Learning Framework for Sparse Neural Networkscited by 15
8 more
  1. 2018Lifted Proximal Operator Machinescited by 14
  2. 2019Convex-Concave Backtracking for Inertial Bregman Proximal Gradient Algorithms in Non-Convex Optimizationcited by 12
  3. 2019Contrastive Learning for Lifted Networkscited by 9
  4. 2017Choose your path wisely: gradient descent in a Bregman distance frameworkcited by 8
  5. 2019Parametric Majorization for Data-Driven Energy Minimization Methodscited by 4
  6. 2020Lifted Regression/Reconstruction Networkscited by 4
  7. 2020Generalised Perceptron Learningcited by 4
  8. 2015Bregman Distances in Inverse Problems and Partial Differential Equationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 0; 2026: 5.

10 papers.

  1. 2024A primal-dual algorithm for image reconstruction with input-convex neural network regularizerscited by 3
  2. 2026Sparse Training of Neural Networks based on Multilevel Mirror Descentcited by 3
  3. 2023A Lifted Bregman Formulation for the Inversion of Deep Neural Networkscited by 2
  4. 2023On Model Compression for Neural Networks: Framework, Algorithm, and Convergence Guaranteecited by 1
  5. 2023Learning a Sparse Representation of Barron Functions with the Inverse Scale Space Flowcited by 1
  6. 2026HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregationcited by 1
  7. 2026Augmented Lagrangian Predictive Codingcited by 1
  8. 2024A lifted Bregman strategy for training unfolded proximal neural network Gaussian denoiserscited by 0
  9. 2026Adaptive Regularization for Sparsity Control in Bregman-Based Optimizerscited by 0
  10. 2026Analog-Friendly Predictive Coding without Activation Derivativescited by 0