mlsanityData to 7 Oct 2026

Neural Optimization Kernel: Towards Robust Deep Learning

arXiv 2106.06097 ↗First posted 11 Jun 2021stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Deep neural networks (NN) have achieved great success in many applications. However, why do deep neural networks obtain good generalization at an over-parameterization regime is still unclear. To better understand deep NN, we establish the connection between deep NN and a novel kernel family, i.e., Neural Optimization Kernel (NOK). The architecture of structured approximation of NOK performs monotonic descent updates of implicit regularization problems. We can implicitly choose the regularization problems by employing different activation functions, e.g., ReLU, max pooling, and soft-thresholding. We further establish a new generalization bound of our deep structured approximated NOK architecture. Our unsupervised structured approximated NOK block can serve as a simple plug-in of popular backbones for a good generalization against input noise.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

29 of 44 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2016Wide Residual Networkscited by 3,057
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  9. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  10. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  11. 2018Feature Denoising for Improving Adversarial Robustnesscited by 312
  12. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
17 more
  1. 2016Deep Learning without Poor Local Minimacited by 233
  2. 2015The Power of Depth for Feedforward Neural Networkscited by 181
  3. 2015Training Very Deep Networkscited by 172
  4. 2019Linearized two-layers neural networks in high dimensioncited by 155
  5. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  6. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  7. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  8. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  9. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  10. 2016Orthogonal Random Featurescited by 89
  11. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 82
  12. 2020Neural Kernels Without Tangentscited by 56
  13. 2020Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regimecited by 35
  14. 2018Learning Two Layer Rectified Neural Networks in Polynomial Timecited by 31
  15. 2017Theoretical properties of the global optimizer of two layer neural networkcited by 3
  16. 2020Subgroup-based Rank-1 Lattice Quasi-Monte Carlocited by 3
  17. 2017Non-Convex Rank/Sparsity Regularization and Local Minimacited by 1

Built on by

Not cited yet.