# Dive into Deep Learning

arXiv: https://arxiv.org/abs/2106.11342 · HTML: https://mlsanity.com/paper/2106.11342/

## Key facts

- Cited by 111 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 187 references that are in the map.
- First posted on arXiv 21 Jun 2021.

## Builds on (most cited first, 30 of 60)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)

## Built on by (most cited first, 30 of 111)

- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations](https://mlsanity.com/paper/2306.04618/) (2023)
- [When Deep Learning Meets Polyhedral Theory: A Survey](https://mlsanity.com/paper/2305.00241/) (2023)
- [How to avoid machine learning pitfalls: a guide for academic researchers](https://mlsanity.com/paper/2108.02497/) (2021)
- [Generalized Huber Loss for Robust Learning and its Efficient Minimization for a Robust Statistics](https://mlsanity.com/paper/2108.12627/) (2021)
- [Applied Causal Inference Powered by ML and AI](https://mlsanity.com/paper/2403.02467/) (2024)
- [Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation](https://mlsanity.com/paper/2403.11808/) (2024)
- [HashAttention: Semantic Sparsity for Faster Inference](https://mlsanity.com/paper/2412.14468/) (2024)
- [DeepSTL -- From English Requirements to Signal Temporal Logic](https://mlsanity.com/paper/2109.10294/) (2021)
- [Theoretical Perspectives on Deep Learning Methods in Inverse Problems](https://mlsanity.com/paper/2206.14373/) (2022)
- [Evaluating ChatGPT as a Recommender System: A Rigorous Approach](https://mlsanity.com/paper/2309.03613/) (2023)
- [LOLA: LLM-Assisted Online Learning Algorithm for Content Experiments](https://mlsanity.com/paper/2406.02611/) (2024)
- [Attention-based Random Forest and Contamination Model](https://mlsanity.com/paper/2201.02880/) (2022)
- [Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction](https://mlsanity.com/paper/2412.17810/) (2024)
- [Hidden Dynamics of Massive Activations in Transformer Training](https://mlsanity.com/paper/2508.03616/) (2025)
- [UEval: A Benchmark for Unified Multimodal Generation](https://mlsanity.com/paper/2601.22155/) (2026)
- [PatchGT: Transformer over Non-trainable Clusters for Learning Graph Representations](https://mlsanity.com/paper/2211.14425/) (2022)
- [Towards Single Camera Human 3D-Kinematics](https://mlsanity.com/paper/2301.05435/) (2023)
- [HalluciDet: Hallucinating RGB Modality for Person Detection Through Privileged Information](https://mlsanity.com/paper/2310.04662/) (2023)
- [Phenomenological Causality](https://mlsanity.com/paper/2211.09024/) (2022)
- [DeepSI: Interactive Deep Learning for Semantic Interaction](https://mlsanity.com/paper/2305.18357/) (2023)
- [Multi-Attention Multiple Instance Learning](https://mlsanity.com/paper/2112.06071/) (2021)
- [Emergency Vehicles Audio Detection and Localization in Autonomous Driving](https://mlsanity.com/paper/2109.14797/) (2021)
- [AGBoost: Attention-based Modification of Gradient Boosting Machine](https://mlsanity.com/paper/2207.05724/) (2022)
- [textitFastSVD-ML-ROM: A Reduced-Order Modeling Framework based on Machine Learning for Real-Time Applications](https://mlsanity.com/paper/2207.11842/) (2022)
- [A Continuous Convolutional Trainable Filter for Modelling Unstructured Data](https://mlsanity.com/paper/2210.13416/) (2022)
- [The Deep Promotion Time Cure Model](https://mlsanity.com/paper/2305.11575/) (2023)
- [The 1st Data Science for Pavements Challenge](https://mlsanity.com/paper/2206.04874/) (2022)
- [Attention and Self-Attention in Random Forests](https://mlsanity.com/paper/2207.04293/) (2022)

## Abstract (arXiv)

This open-source book represents our attempt to make deep learning approachable, teaching readers the concepts, the context, and the code. The entire book is drafted in Jupyter notebooks, seamlessly integrating exposition figures, math, and interactive examples with self-contained code. Our goal is to offer a resource that could (i) be freely available for everyone; (ii) offer sufficient technical depth to provide a starting point on the path to actually becoming an applied machine learning scientist; (iii) include runnable code, showing readers how to solve problems in practice; (iv) allow for rapid updates, both by us and also by the community at large; (v) be complemented by a forum for interactive discussion of technical details and to answer questions.