# A Kernel-Expanded Stochastic Neural Network

arXiv: https://arxiv.org/abs/2201.05319 · HTML: https://mlsanity.com/paper/2201.05319/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 69 references that are in the map.
- First posted on arXiv 14 Jan 2022.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Adding Gradient Noise Improves Learning for Very Deep Networks](https://mlsanity.com/paper/1511.06807/) (2015)
- [Underdamped Langevin MCMC: A non-asymptotic analysis](https://mlsanity.com/paper/1707.03663/) (2017)
- [An Improved Analysis of Training Over-parameterized Deep Neural Networks](https://mlsanity.com/paper/1906.04688/) (2019)
- [The loss surface of deep and wide neural networks](https://mlsanity.com/paper/1704.08045/) (2017)
- [Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization](https://mlsanity.com/paper/1710.05179/) (2017)
- [Noisy Activation Functions](https://mlsanity.com/paper/1603.00391/) (2016)
- [Convolutional Kernel Networks](https://mlsanity.com/paper/1406.3332/) (2014)
- [Statistical consistency and asymptotic normality for high-dimensional robust M-estimators](https://mlsanity.com/paper/1501.00312/) (2015)
- [Consistent Sparse Deep Learning: Theory and Computation](https://mlsanity.com/paper/2102.13229/) (2021)
- [A representer theorem for deep kernel learning](https://mlsanity.com/paper/1709.10441/) (2017)
- [Support recovery without incoherence: A case for nonconvex regularization](https://mlsanity.com/paper/1412.5632/) (2014)
- [Adversarial Noise Layer: Regularize Neural Network By Adding Noise](https://mlsanity.com/paper/1805.08000/) (2018)
- [Uncertainty Quantification for Sparse Deep Learning](https://mlsanity.com/paper/2002.11815/) (2020)
- [Deep Multiple Kernel Learning](https://mlsanity.com/paper/1310.3101/) (2013)

## Built on by (most cited first, 9 of 9)

- [Nonlinear Sufficient Dimension Reduction with a Stochastic Neural Network](https://mlsanity.com/paper/2210.04349/) (2022)
- [Belted and Ensembled Neural Network for Linear and Nonlinear Sufficient Dimension Reduction](https://mlsanity.com/paper/2412.08961/) (2024)
- [Causal-StoNet: Causal Inference for High-Dimensional Complex Data](https://mlsanity.com/paper/2403.18994/) (2024)
- [Uncertainty Quantification for Physics-Informed Neural Networks with Extended Fiducial Inference](https://mlsanity.com/paper/2505.19136/) (2025)
- [Density Regression and Uncertainty Quantification with Bayesian Deep Noise Neural Networks](https://mlsanity.com/paper/2206.05643/) (2022)
- [Magnitude Pruning of Large Pretrained Transformer Models with a Mixture Gaussian Prior](https://mlsanity.com/paper/2411.00969/) (2024)
- [Extended Fiducial Inference for Individual Treatment Effects via Deep Neural Networks](https://mlsanity.com/paper/2505.01995/) (2025)
- [Uncertainty Quantification for Large-Scale Deep Networks via Post-StoNet Modeling](https://mlsanity.com/paper/2508.01217/) (2025)
- [Sublinearly Structured Deep Neural Networks Achieve Feature Learning Consistency for Compositional Functions](https://mlsanity.com/paper/2606.23477/) (2026)

## Abstract (arXiv)

The deep neural network suffers from many fundamental issues in machine learning. For example, it often gets trapped into a local minimum in training, and its prediction uncertainty is hard to be assessed. To address these issues, we propose the so-called kernel-expanded stochastic neural network (K-StoNet) model, which incorporates support vector regression (SVR) as the first hidden layer and reformulates the neural network as a latent variable model. The former maps the input vector into an infinite dimensional feature space via a radial basis function (RBF) kernel, ensuring absence of local minima on its training loss surface. The latter breaks the high-dimensional nonconvex neural network training problem into a series of low-dimensional convex optimization problems, and enables its prediction uncertainty easily assessed. The K-StoNet can be easily trained using the imputation-regularized optimization (IRO) algorithm. Compared to traditional deep neural networks, K-StoNet possesses a theoretical guarantee to asymptotically converge to the global optimum and enables the prediction uncertainty easily assessed. The performances of the new model in training, prediction and uncertainty quantification are illustrated by simulated and real data examples.