mlsanityData to 14 Oct 2026

Convex Dual Theory Analysis of Two-Layer Convolutional Neural Networks with Soft-Thresholding

arXiv 2304.06959 ↗First posted 14 Apr 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Soft-thresholding has been widely used in neural networks. Its basic network structure is a two-layer convolution neural network with soft-thresholding. Due to the network's nature of nonlinearity and nonconvexity, the training process heavily depends on an appropriate initialization of network parameters, resulting in the difficulty of obtaining a globally optimal solution. To address this issue, a convex dual network is designed here. We theoretically analyze the network convexity and numerically confirm that the strong duality holds. This conclusion is further verified in the linear fitting and denoising experiments. This work provides a new way to convexify soft-thresholding neural networks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 44 of its 93 references that are in the map. First posted on arXiv 14 Apr 2023.

Builds on

44 of 93 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2014Going Deeper with Convolutionscited by 4,241
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  9. 2021Convex optimizationcited by 2,515
  10. 2020On Layer Normalization in the Transformer Architecturecited by 922
  11. 2017Deep Reinforcement Learning that Matterscited by 754
  12. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
32 more
  1. 2016Input Convex Neural Networkscited by 414
  2. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  3. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  4. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
  5. 2020Understanding the Difficulty of Training Transformerscited by 210
  6. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  7. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 174
  8. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  9. 2019How do infinite width bounded norm networks look in function space?cited by 106
  10. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 98
  11. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 88
  12. 2017On weight initialization in deep neural networkscited by 67
  13. 2020Convex Geometry and Duality of Over-parameterized Neural Networkscited by 49
  14. 2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
  15. 2017Geometry of Optimization and Implicit Regularization in Deep Learningcited by 39
  16. 2020Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithmscited by 34
  17. 2021WINNet: Wavelet-inspired Invertible Network for Image Denoisingcited by 34
  18. 2022Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositionscited by 31
  19. 2021On the Reproducibility of Neural Network Predictionscited by 29
  20. 2021One-dimensional Deep Low-rank and Sparse Network for Accelerated MRIcited by 27
  21. 2021Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programscited by 25
  22. 2022Physics-driven Synthetic Data Learning for Biomedical Magnetic Resonancecited by 22
  23. 2021Accelerated MRI Reconstruction with Separable and Enhanced Low-Rank Hankel Regularizationcited by 18
  24. 2019Accelerated Nuclear Magnetic Resonance Spectroscopy with Deep Learningcited by 16
  25. 2020Convex Regularization Behind Neural Reconstructioncited by 15
  26. 2020A Sparse Model-inspired Deep Thresholding Network for Exponential Signal Reconstruction -- Application in Fast Biological Spectroscopycited by 12
  27. 2021The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Programcited by 11
  28. 2019pISTA-SENSE-ResNet for Parallel MRI Reconstructioncited by 10
  29. 2020Exponential Signal Reconstruction with Deep Hankel Matrix Factorizationcited by 8
  30. 2018Nonlocality-Reinforced Convolutional Neural Networks for Image Denoisingcited by 6
  31. 2020Implicit Convex Regularizers of CNN Architectures: Convex Optimization of Two- and Three-Layer Networks in Polynomial Timecited by 4
  32. 2020Review and Prospect: NMR Spectroscopy Denoising & Reconstruction with Low Rank Hankel Matrices and Tensorscited by 3

Built on by

Not cited yet.