# Merging Two Cultures: Deep and Statistical Learning

arXiv: https://arxiv.org/abs/2110.11561 · HTML: https://mlsanity.com/paper/2110.11561/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 116 references that are in the map.
- First posted on arXiv 22 Oct 2021.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](https://mlsanity.com/paper/1909.06312/) (2019)
- [Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review](https://mlsanity.com/paper/1611.00740/) (2016)
- [Posterior Concentration for Sparse Deep Learning](https://mlsanity.com/paper/1803.09138/) (2018)
- [Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors](https://mlsanity.com/paper/1806.05975/) (2018)
- [A Selective Overview of Deep Learning](https://mlsanity.com/paper/1904.05526/) (2019)
- [Risk-Averse Bayes-Adaptive Reinforcement Learning](https://mlsanity.com/paper/2102.05762/) (2021)
- [Deep Learning: A Bayesian Perspective](https://mlsanity.com/paper/1706.00473/) (2017)
- [Bayesian Recurrent Neural Network Models for Forecasting and Quantifying Uncertainty in Spatial-Temporal Data](https://mlsanity.com/paper/1711.00636/) (2017)
- [The Deep Learning Revolution and Its Implications for Computer Architecture and Chip Design](https://mlsanity.com/paper/1911.05289/) (2019)
- [Uncertainty Quantification for Sparse Deep Learning](https://mlsanity.com/paper/2002.11815/) (2020)
- [Deep Learning for Spatio-Temporal Modeling: Dynamic Traffic Flows and High Frequency Trading](https://mlsanity.com/paper/1705.09851/) (2017)
- [Horseshoe Regularization for Machine Learning in Complex and Deep Models](https://mlsanity.com/paper/1904.10939/) (2019)
- [Deep Learning Partial Least Squares](https://mlsanity.com/paper/2106.14085/) (2021)
- [From Least Squares to Signal Processing and Particle Filtering](https://mlsanity.com/paper/1705.04141/) (2017)

## Built on by (most cited first, 4 of 4)

- [Quantum Bayesian Computation](https://mlsanity.com/paper/2208.08068/) (2022)
- [Generative Modeling: A Review](https://mlsanity.com/paper/2501.05458/) (2024)
- [Kolmogorov GAM Networks are all you need!](https://mlsanity.com/paper/2501.00704/) (2025)
- [Generative Bayesian Hyperparameter Tuning](https://mlsanity.com/paper/2512.20051/) (2025)

## Abstract (arXiv)

Merging the two cultures of deep and statistical learning provides insights into structured high-dimensional data. Traditional statistical modeling is still a dominant strategy for structured tabular data. Deep learning can be viewed through the lens of generalized linear models (GLMs) with composite link functions. Sufficient dimensionality reduction (SDR) and sparsity performs nonlinear feature engineering. We show that prediction, interpolation and uncertainty quantification can be achieved using probabilistic methods at the output layer of the model. Thus a general framework for machine learning arises that first generates nonlinear features (a.k.a factors) via sparse regularization and stochastic gradient optimisation and second uses a stochastic output layer for predictive uncertainty. Rather than using shallow additive architectures as in many statistical models, deep learning uses layers of semi affine input transformations to provide a predictive rule. Applying these layers of transformations leads to a set of attributes (a.k.a features) to which predictive statistical methods can be applied. Thus we achieve the best of both worlds: scalability and fast predictive rule construction together with uncertainty quantification. Sparse regularisation with un-supervised or supervised learning finds the features. We clarify the duality between shallow and wide models such as PCA, PPR, RRR and deep but skinny architectures such as autoencoders, MLPs, CNN, and LSTM. The connection with data transformations is of practical importance for finding good network architectures. By incorporating probabilistic components at the output level we allow for predictive uncertainty. For interpolation we use deep Gaussian process and ReLU trees for classification. We provide applications to regression, classification and interpolation. Finally, we conclude with directions for future research.