# Why is AI hard and Physics simple?

arXiv: https://arxiv.org/abs/2104.00008 · HTML: https://mlsanity.com/paper/2104.00008/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 81 references that are in the map.
- First posted on arXiv 31 Mar 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [On the Expressive Power of Deep Neural Networks](https://mlsanity.com/paper/1606.05336/) (2016)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [Deep Information Propagation](https://mlsanity.com/paper/1611.01232/) (2016)
- [Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks](https://mlsanity.com/paper/1907.04595/) (2019)
- [Building machines that adapt and compute like brains](https://mlsanity.com/paper/1711.04203/) (2017)
- [Non-Gaussian processes and neural networks at finite widths](https://mlsanity.com/paper/1910.00019/) (2019)

## Built on by (most cited first, 7 of 7)

- [The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers](https://mlsanity.com/paper/2210.06313/) (2022)
- [The edge of chaos: quantum field theory and deep neural networks](https://mlsanity.com/paper/2109.13247/) (2021)
- [Neural Scaling Laws Rooted in the Data Distribution](https://mlsanity.com/paper/2412.07942/) (2024)
- [Structures of Neural Network Effective Theories](https://mlsanity.com/paper/2305.02334/) (2023)
- [Laziness, Barren Plateau, and Noise in Machine Learning](https://mlsanity.com/paper/2206.09313/) (2022)
- [Gauge-covariant stochastic neural fields: Stability and finite-width effects](https://mlsanity.com/paper/2508.18948/) (2025)
- [TetrisCNN for interpretable detection of phases of matter from experimental quantum simulator data](https://mlsanity.com/paper/2609.20693/) (2026)

## Abstract (arXiv)

We discuss why AI is hard and why physics is simple. We discuss how physical intuition and the approach of theoretical physics can be brought to bear on the field of artificial intelligence and specifically machine learning. We suggest that the underlying project of machine learning and the underlying project of physics are strongly coupled through the principle of sparsity, and we call upon theoretical physicists to work on AI as physicists. As a first step in that direction, we discuss an upcoming book on the principles of deep learning theory that attempts to realize this approach.