mlsanityData to 7 Oct 2026

Training Quantized Neural Networks to Global Optimality via Semidefinite Programming

arXiv 2105.01420 ↗First posted 4 May 2021cs.LG stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

Neural networks (NNs) have been extremely successful across many tasks in machine learning. Quantization of NN weights has become an important topic due to its impact on their energy efficiency, inference time and deployment on hardware. Although post-training quantization is well-studied, training optimal quantized NNs involves combinatorial non-convex optimization problems which appear intractable. In this work, we introduce a convex optimization strategy to train quantized NNs with polynomial activations. Our method leverages hidden convexity in two-layer neural networks from the recent literature, semidefinite lifting, and Grothendieck's identity. Surprisingly, we show that certain quantized NN problems can be solved to global optimality in polynomial-time in all relevant parameters via semidefinite relaxations. We present numerical examples to illustrate the effectiveness of our method.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 25 of its 41 references that are in the map. First posted on arXiv 4 May 2021.

Builds on

25 of 41 references.

References · 25

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  3. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 633
  4. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  5. 2015Deep Learning with Limited Numerical Precisioncited by 368
  6. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  7. 2016Trained Ternary Quantizationcited by 203
  8. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
  9. 2017A Rewriting System for Convex Optimization Problemscited by 178
  10. 2015Fixed Point Quantization of Deep Convolutional Networkscited by 141
  11. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 96
  12. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
13 more
  1. 2019Scalable Semidefinite Programmingcited by 36
  2. 2020Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithmscited by 33
  3. 2021Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularizationcited by 21
  4. 2020Convex Regularization Behind Neural Reconstructioncited by 15
  5. 2021Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Timecited by 14
  6. 2017Sketchy Decisions: Convex Low-Rank Matrix Optimization with Optimal Storagecited by 11
  7. 2020Optimal Randomized First-Order Methods for Least-Squares Problemscited by 10
  8. 2016Overcoming Challenges in Fixed Point Training of Deep Convolutional Networkscited by 9
  9. 2020Effective Dimension Adaptive Sketching Methods for Faster Regularized Least-Squares Optimizationcited by 9
  10. 2020Adaptive and Oblivious Randomized Subspace Methods for High-Dimensional Optimization: Sharp Analysis and Lower Boundscited by 6
  11. 2020Optimal Iterative Sketching with the Subsampled Randomized Hadamard Transformcited by 3
  12. 2021Fast Convex Quadratic Optimization Solvers with Adaptive Sketching-based Preconditionerscited by 3
  13. 2018Convex Relaxations of Convolutional Neural Netscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 2; 2024: 0; 2025: 0; 2026: 1.

6 papers.

  1. 2022Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformerscited by 30
  2. 2021The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Programcited by 11
  3. 2022Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementationcited by 6
  4. 2023From Complexity to Clarity: Analytical Expressions of Deep Neural Network Weights via Clifford's Geometric Algebra and Convexitycited by 4
  5. 2023MixQuant: Mixed Precision Quantization with a Bit-width Optimization Searchcited by 4
  6. 2026Exact Convex Reformulations of Linear Neural Networks via Completely Positive Liftingcited by 0