mlsanityData to 7 Oct 2026

Model Complexity of Deep Learning: A Survey

arXiv 2103.05127 ↗First posted 8 Mar 2021cs.LG cs.AIcited by 64

Not on the map. Why?

Abstract (arXiv)

Model complexity is a fundamental problem in deep learning. In this paper we conduct a systematic overview of the latest studies on model complexity in deep learning. Model complexity of deep learning can be categorized into expressive capacity and effective model complexity. We review the existing studies on those two categories along four important factors, including model framework, model size, optimization process and data complexity. We also discuss the applications of deep learning model complexity including understanding model generalization, model optimization, and model selection and design. We conclude by proposing several interesting future directions.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Builds on 46 of its 114 references that are in the map. First posted on arXiv 8 Mar 2021.

Builds on

46 of 114 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  6. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  9. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  10. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  11. 2017Exploring Generalization in Deep Learningcited by 595
  12. 2017Progressive Neural Architecture Searchcited by 483
34 more
  1. 2013Do Deep Nets Really Need to be Deep?cited by 466
  2. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  3. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  4. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  5. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
  6. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  7. 2016On the Expressive Power of Deep Neural Networkscited by 302
  8. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  9. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  10. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 253
  11. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  12. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  13. 2013Maxout Networkscited by 186
  14. 2019Complexity of Linear Regions in Deep Networkscited by 166
  15. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  16. 2017Generalization in Deep Learningcited by 146
  17. 2018Phrase-Based & Neural Unsupervised Machine Translationcited by 138
  18. 2018Rademacher Complexity for Adversarially Robust Generalizationcited by 132
  19. 2018Activation Functions: Comparison of trends in Practice and Research for Deep Learningcited by 130
  20. 2017Approximating Continuous Functions by ReLU Nets of Minimal Widthcited by 120
  21. 2017Fisher-Rao Metric, Geometry, and Complexity of Neural Networkscited by 99
  22. 2017Implicit Regularization in Deep Learningcited by 90
  23. 2019Optimization for deep learning: theory and algorithmscited by 70
  24. 2019On Network Design Spaces for Visual Recognitioncited by 68
  25. 2019On the Expressive Power of Deep Polynomial Neural Networkscited by 65
  26. 2016Convolutional Rectifier Networks as Generalized Tensor Decompositionscited by 32
  27. 2017Theory of Deep Learning III: explaining the non-overfitting puzzlecited by 32
  28. 2017Expressive power of recurrent neural networkscited by 30
  29. 2016Learning Functions: When Is Deep Better Than Shallowcited by 19
  30. 2018On the Selection of Initialization and Activation Function for Deep Neural Networkscited by 11
  31. 2018Capacity Control of ReLU Neural Networks by Basis-path Normcited by 10
  32. 2020Measuring Model Complexity of Neural Networks with Curve Activation Functionscited by 5
  33. 2020Learning with tree tensor networks: complexity estimates and model selectioncited by 5
  34. 2019On the complexity of logistic regression modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 14; 2024: 23; 2025: 16; 2026: 1.

64 papers.

Sort:

  1. 2021Trustworthy AI: From Principles to Practicescited by 206
  2. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  3. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  4. 2023Evidence of interrelated cognitive-like capabilities in large language models: Indications of artificial general intelligence or achievement?cited by 26
  5. 2023Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunitiescited by 21
  6. 2022Domain-incremental Cardiac Image Segmentation with Style-oriented Replay and Domain-sensitive Feature Whiteningcited by 16
  7. 2022Backbones-Review: Feature Extraction Networks for Deep Learning and Deep Reinforcement Learning Approachescited by 15
  8. 2023Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexitycited by 13
  9. 2024See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decompositioncited by 13
  10. 2024A Scoping Review of Deep Learning Methods for Photoplethysmography Datacited by 11
  11. 2021Interference Suppression Using Deep Learning: Current Approaches and Open Challengescited by 10
  12. 2023Bug Characterization in Machine Learning-based Systemscited by 10
38 more
  1. 2023Agglomerative Federated Learning: Empowering Larger Model Training via End-Edge-Cloud Collaborationcited by 10
  2. 2021A Novel Intrinsic Measure of Data Separabilitycited by 9
  3. 2022From Modern CNNs to Vision Transformers: Assessing the Performance, Robustness, and Classification Strategies of Deep Learning Models in Histopathologycited by 9
  4. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
  5. 2025Benchmarking Quantum Reinforcement Learningcited by 9
  6. 2024Boosting Adversarial Training via Fisher-Rao Norm-based Regularizationcited by 7
  7. 2024Transformer-based Image and Video Inpainting: Current Challenges and Future Directionscited by 6
  8. 2023Deep ReLU Networks Have Surprisingly Simple Polytopescited by 5
  9. 2024Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuningcited by 4
  10. 2024Study on the Helpfulness of Explainable Artificial Intelligencecited by 4
  11. 2024RedTest: Towards Measuring Redundancy in Deep Neural Networks Effectivelycited by 4
  12. 2025The role of data partitioning on the performance of EEG-based deep learning models in supervised cross-subject analysis: a preliminary studycited by 4
  13. 2022Model-based Deep Learning Receiver Design for Rate-Splitting Multiple Accesscited by 3
  14. 2024A Two-Scale Complexity Measure for Deep Learning Modelscited by 3
  15. 2024Cracking Factual Knowledge: A Comprehensive Analysis of Degenerate Knowledge Neurons in Large Language Modelscited by 3
  16. 2024Explainable AI Integrated Feature Engineering for Wildfire Predictioncited by 3
  17. 2025MSA-CNN: A Lightweight Multi-Scale CNN with Attention for Sleep Stage Classificationcited by 3
  18. 2025Deep Learning-based Techniques for Integrated Sensing and Communication Systems: State-of-the-Art, Challenges, and Opportunitiescited by 3
  19. 2023Synaptic Stripping: How Pruning Can Bring Dead Neurons Back To Lifecited by 2
  20. 2023Good-looking but Lacking Faithfulness: Understanding Local Explanation Methods through Trend-based Testingcited by 2
  21. 2024Generalization and Risk Bounds for Recurrent Neural Networkscited by 2
  22. 2025Generating High-Quality Datasets for Code Editing via Open-Source Language Modelscited by 2
  23. 2022CourtNet for Infrared Small-Target Detectioncited by 1
  24. 2023Pathway to a fully data-driven geotechnics: lessons from materials informaticscited by 1
  25. 2024Ensembler: Protect Collaborative Inference Privacy from Model Inversion Attack via Selective Ensemblecited by 1
  26. 2024A Computational Tropical Geometry Framework for Neural Networkscited by 1
  27. 2024The impact of model size on catastrophic forgetting in Online Continual Learningcited by 1
  28. 2024CAMAL: Optimizing LSM-trees via Active Learningcited by 1
  29. 2025Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphscited by 1
  30. 2025Adaptive Classification of Interval-Valued Time Seriescited by 1
  31. 2025Cross-Branch Orthogonality for Improved Generalization in Face Deepfake Detectioncited by 1
  32. 2025Energy Considerations for Large Pretrained Neural Networkscited by 1
  33. 2025Adaptive Neural Quantum States: A Recurrent Neural Network Perspectivecited by 1
  34. 2025On Optimal Hyperparameters for Differentially Private Deep Transfer Learningcited by 1
  35. 2026Quantum Circuit-Based Learning Models: Bridging Quantum Computing and Machine Learningcited by 1
  36. 2022Rethinking Saliency Map: An Context-aware Perturbation Method to Explain EEG-based Deep Learning Modelcited by 0
  37. 2023Revisiting Sample Size Determination in Natural Language Understandingcited by 0
  38. 2023Transformers for Green Semantic Communication: Less Energy, More Semanticscited by 0

50 most cited of 64 All 64 (JSON)