mlsanityData to 7 Oct 2026

A Survey of Deep Learning: From Activations to Transformers

arXiv 2302.00722 ↗First posted 1 Feb 2023cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Deep learning has made tremendous progress in the last decade. A key success factor is the large amount of architectures, layers, objectives, and optimization techniques. They include a myriad of variants related to attention, normalization, skip connections, transformers and self-supervised learning schemes -- to name a few. We provide a comprehensive overview of the most important, recent works in these areas to those who already have a basic understanding of deep learning. We hope that a holistic and unified treatment of influential, recent works helps researchers to form new connections between diverse areas of deep learning. We identify and discuss multiple patterns that summarize the key strategies for many of the successful innovations over the last decade as well as works that can be seen as rising stars. We also include a discussion on recent commercially built, closed-source models such as OpenAI's GPT-4 and Google's PaLM 2.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 61 of its 96 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

61 of 96 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  12. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
38 more
  1. 2017Graph Attention Networkscited by 8,854
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2016Layer Normalizationcited by 5,579
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  13. 2020Longformer: The Long-Document Transformercited by 3,362
  14. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  15. 2020Supervised Contrastive Learningcited by 2,803
  16. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  17. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  18. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  19. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  20. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  21. 2016node2vec: Scalable Feature Learning for Networkscited by 1,913
  22. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  23. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  24. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  25. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  26. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  27. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  28. 2021Transformers in Vision: A Surveycited by 1,009
  29. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  30. 2019RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Spacecited by 893
  31. 2019On the Convergence of Adam and Beyondcited by 834
  32. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  33. 2021Going deeper with Image Transformerscited by 759
  34. 2018Which Training Methods for GANs do actually Converge?cited by 672
  35. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  36. 2020A Generalization of Transformer Networks to Graphscited by 653
  37. 2020A Survey on Visual Transformercited by 653
  38. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 2.

5 papers.

  1. 2023Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directionscited by 123
  2. 2025Large language models for automated scholarly paper review: A surveycited by 59
  3. 2024What comes after transformers? -- A selective survey connecting ideas in deep learningcited by 3
  4. 2024Topic Modeling with Fine-tuning LLMs and Bag of Sentencescited by 2
  5. 2025S-Crescendo: A Nested Transformer Weaving Framework for Scalable Nonlinear System in S-Domain Representationcited by 0