# Perspectives on the State and Future of Deep Learning - 2023

arXiv: https://arxiv.org/abs/2312.09323 · HTML: https://mlsanity.com/paper/2312.09323/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Builds on (most cited first, 27 of 27)

- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data](https://mlsanity.com/paper/1703.11008/) (2017)
- [Just Train Twice: Improving Group Robustness without Training Group Information](https://mlsanity.com/paper/2107.09044/) (2021)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups](https://mlsanity.com/paper/2104.09459/) (2021)
- [Practical Deep Learning with Bayesian Principles](https://mlsanity.com/paper/1906.02506/) (2019)
- [A Bayesian Perspective on Generalization and Stochastic Gradient Descent](https://mlsanity.com/paper/1710.06451/) (2017)
- [Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach](https://mlsanity.com/paper/1804.05862/) (2018)
- [PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization](https://mlsanity.com/paper/2211.13609/) (2022)
- [Residual Pathway Priors for Soft Equivariance Constraints](https://mlsanity.com/paper/2112.01388/) (2021)
- [Stochastic Training is Not Necessary for Generalization](https://mlsanity.com/paper/2109.14119/) (2021)
- [The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning](https://mlsanity.com/paper/2304.05366/) (2023)
- [Understanding Generalization through Visualizations](https://mlsanity.com/paper/1906.03291/) (2019)
- [Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited](https://mlsanity.com/paper/2003.02139/) (2020)
- [How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization](https://mlsanity.com/paper/2210.06441/) (2022)
- [Dangers of Bayesian Model Averaging under Covariate Shift](https://mlsanity.com/paper/2106.11905/) (2021)
- [CoLA: Exploiting Compositional Structure for Automatic and Efficient Numerical Linear Algebra](https://mlsanity.com/paper/2309.03060/) (2023)

## Built on by (most cited first, 4 of 4)

- [Calibrating Large Language Models Using Their Generations Only](https://mlsanity.com/paper/2403.05973/) (2024)
- [Better, Not Just More: Data-Centric Machine Learning for Earth Observation](https://mlsanity.com/paper/2312.05327/) (2023)
- [Spin glass model of in-context learning](https://mlsanity.com/paper/2408.02288/) (2024)
- [Gaussian Universality for Diffusion Models](https://mlsanity.com/paper/2501.07741/) (2025)

## Abstract (arXiv)

The goal of this series is to chronicle opinions and issues in the field of machine learning as they stand today and as they change over time. The plan is to host this survey periodically until the AI singularity paperclip-frenzy-driven doomsday, keeping an updated list of topical questions and interviewing new community members for each edition. In this issue, we probed people's opinions on interpretable AI, the value of benchmarking in modern NLP, the state of progress towards understanding deep learning, and the future of academia.