mlsanityData to 7 Oct 2026

Globally Optimal Training of Neural Networks with Threshold Activation Functions

arXiv 2303.03382 ↗First posted 6 Mar 2023cs.LG stat.MLcited by 9

Not on the map. Why?

Abstract (arXiv)

Threshold activation functions are highly preferable in neural networks due to their efficiency in hardware implementations. Moreover, their mode of operation is more interpretable and resembles that of biological neurons. However, traditional gradient based algorithms such as Gradient Descent cannot be used to train the parameters of neural networks with threshold activations since the activation function has zero gradient except at a single non-differentiable point. To this end, we study weight decay regularized training problems of deep neural networks with threshold activations. We first show that regularized deep threshold network training problems can be equivalently formulated as a standard convex optimization problem, which parallels the LASSO method, provided that the last hidden layer width exceeds a certain threshold. We also derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network. We corroborate our theoretical results with various numerical experiments.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 26 of its 46 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

26 of 46 references.

References · 26

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2021Convex optimizationcited by 2,479
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  4. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  5. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  6. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  7. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
  8. 2015Compressing Neural Networks with the Hashing Trickcited by 150
  9. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 115
  10. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 96
  11. 2019Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimizationcited by 58
  12. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
14 more
  1. 2020Convex Geometry and Duality of Over-parameterized Neural Networkscited by 49
  2. 2020Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithmscited by 33
  3. 2022Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformerscited by 30
  4. 2021Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programscited by 25
  5. 2021Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularizationcited by 21
  6. 2018Blended Coarse Gradient Descent for Full Quantization of Deep Neural Networkscited by 20
  7. 2020Convex Regularization Behind Neural Reconstructioncited by 15
  8. 2021Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Timecited by 14
  9. 2021Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutionscited by 14
  10. 2021Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networkscited by 14
  11. 2020Memory capacity of neural networks with threshold and ReLU activationscited by 13
  12. 2022Convexifying Transformers: Improving optimization and understanding of transformer networkscited by 12
  13. 2021Parallel Deep Neural Networks Have Zero Duality Gapcited by 9
  14. 2020On Sparsity in Overparametrised Shallow ReLU Networkscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 0; 2026: 3.

9 papers.

  1. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  2. 2024Analyzing Neural Network-Based Generative Diffusion Models through Convex Optimizationcited by 7
  3. 2023From Complexity to Clarity: Analytical Expressions of Deep Neural Network Weights via Clifford's Geometric Algebra and Convexitycited by 4
  4. 2024Convex Approximation of Two-Layer ReLU Networks for Hidden State Differential Privacycited by 2
  5. 2023Fixing the NTK: From Neural Network Linearizations to Exact Convex Programscited by 1
  6. 2024A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Featurescited by 1
  7. 2026Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuningcited by 0
  8. 2026Globally Optimal Training of Spiking Neural Networks via Parameter Reconstructioncited by 0
  9. 2026A Composite Activation Function for Learning Stable Binary Representationscited by 0