mlsanityData to 7 Oct 2026

Do Deep Nets Really Need to be Deep?

arXiv 1312.6184 ↗First posted 21 Dec 2013cs.LG cs.NEcited by 466

Not on the map. Why?

Abstract (arXiv)

Currently, deep neural networks are the state of the art on problems such as speech recognition and computer vision. In this extended abstract, we show that shallow feed-forward networks can learn the complex functions previously learned by deep nets and achieve accuracies previously only achievable with deep models. Moreover, in some cases the shallow neural nets can learn these deep functions using a total number of parameters similar to the original deep model. We evaluate our method on the TIMIT phoneme recognition task and are able to train shallow fully-connected nets that perform similarly to complex, well-engineered, deep convolutional architectures. Our success in training shallow neural nets to mimic deeper models suggests that there probably exist better algorithms for training shallow feed-forward nets than those currently available.

Key facts. Cited by 466 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Dec 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 139; 2022: 103; 2023: 98; 2024: 58; 2025: 31; 2026: 36.

466 papers.

Sort:

By area:

  1. 2024Depth Anything V2cited by 1,385
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  3. 2022Large Language Models Can Self-Improvecited by 656
  4. 2021ImageNet-21K Pretraining for the Massescited by 607
  5. 2023LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPScited by 420
  6. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  7. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
  8. 2021NeRV: Neural Representations for Videoscited by 255
  9. 2021Does Knowledge Distillation Really Work?cited by 187
  10. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165
  11. 2021Towards Understanding Knowledge Distillationcited by 158
  12. 20222DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Cloudscited by 157
38 more
  1. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 155
  2. 2024Distilling System 2 into System 1cited by 152
  3. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 140
  4. 2022Efficient Few-Shot Learning Without Promptscited by 135
  5. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  6. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  7. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
  8. 2023CompGS: Smaller and Faster Gaussian Splatting with Vector Quantizationcited by 127
  9. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  10. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 120
  11. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  12. 2021Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challengescited by 114
  13. 2022Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidationcited by 113
  14. 2021Dynamic Slimmable Networkcited by 83
  15. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  16. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  17. 2022Knowledge Distillation via the Target-aware Transformercited by 70
  18. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillationcited by 69
  19. 2022R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesiscited by 69
  20. 2021Improving Deep Learning Interpretability by Saliency Guided Trainingcited by 68
  21. 2022Compressing Volumetric Radiance Fields to 1 MBcited by 66
  22. 2021Model Complexity of Deep Learning: A Surveycited by 64
  23. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  24. 2021Distilling Audio-Visual Knowledge by Compositional Contrastive Learningcited by 53
  25. 2021A Survey on Green Deep Learningcited by 52
  26. 2022Semi-Supervised and Unsupervised Deep Visual Learning: A Surveycited by 52
  27. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  28. 2021Deep Model Intellectual Property Protection via Deep Watermarkingcited by 47
  29. 2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 46
  30. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  31. 2024A Survey on Transformer Compressioncited by 44
  32. 2022A Survey on Heterogeneous Federated Learningcited by 42
  33. 2022Cross-Architecture Knowledge Distillationcited by 40
  34. 2021Confidence-Aware Multi-Teacher Knowledge Distillationcited by 39
  35. 2021Distilling Object Detectors with Feature Richnesscited by 38
  36. 2021Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Modelcited by 37
  37. 2021Learning Compatible Embeddingscited by 37
  38. 2022Submodularity In Machine Learning and Artificial Intelligencecited by 37

50 most cited of 466 All 466 (JSON)