mlsanityData to 7 Oct 2026

Mathematical Challenges in Deep Learning

arXiv 2303.15464 ↗First posted 24 Mar 2023cs.LG cs.AI math.STcited by 0

Not on the map. Why?

Abstract (arXiv)

Deep models are dominating the artificial intelligence (AI) industry since the ImageNet challenge in 2012. The size of deep models is increasing ever since, which brings new challenges to this field with applications in cell phones, personal computers, autonomous cars, and wireless base stations. Here we list a set of problems, ranging from training, inference, generalization bound, and optimization with some formalism to communicate these challenges with mathematicians, statisticians, and theoretical computer scientists. This is a subjective view of the research questions in deep learning that benefits the tech industry in long run.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 52 of its 87 references that are in the map. First posted on arXiv 24 Mar 2023.

Builds on

52 of 87 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  4. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  5. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  8. 2019On the Convergence of Adam and Beyondcited by 834
  9. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  10. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  11. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  12. 2016Learning to learn by gradient descent by gradient descentcited by 616
38 more
  1. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  2. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  3. 2017Large Batch Training of Convolutional Networkscited by 529
  4. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  5. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  6. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  7. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
  8. 2019Bridging Theory and Algorithm for Domain Adaptationcited by 332
  9. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  10. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  11. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  12. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 273
  13. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  14. 2016Trained Ternary Quantizationcited by 203
  15. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 183
  16. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
  17. 2020Domain Adaptation with Conditional Distribution Matching and Generalized Label Shiftcited by 124
  18. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 121
  19. 2021The Causal-Neural Connection: Expressiveness, Learnability, and Inferencecited by 97
  20. 2018Training and Inference with Integers in Deep Neural Networkscited by 84
  21. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 75
  22. 2018The Description Length of Deep Learning Modelscited by 71
  23. 2021Intrinsic Dimension, Persistent Homology and Generalization in Neural Networkscited by 70
  24. 2019On-Device Machine Learning: An Algorithms and Learning Theory Perspectivecited by 63
  25. 2015Provable approximation properties for deep neural networkscited by 60
  26. 2018Deep Neural Networks Learn Non-Smooth Functions Effectivelycited by 57
  27. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  28. 2021f-Domain-Adversarial Learning: Theory and Algorithmscited by 47
  29. 2015Explaining the Success of AdaBoost and Random Forests as Interpolating Classifierscited by 45
  30. 2018Mixed Precision Training of Convolutional Neural Networks using Integer Operationscited by 45
  31. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  32. 2021Towards a Unified Information-Theoretic Framework for Generalizationcited by 29
  33. 2021Understanding the Generalization Benefit of Model Invariance from a Data Perspectivecited by 22
  34. 2020NITI: Training Integer Neural Networks Using Integer-only Arithmeticcited by 18
  35. 2020Advantage of Deep Neural Networks for Estimating Functions with Singularity on Hypersurfacescited by 17
  36. 2022Low-Precision Stochastic Gradient Langevin Dynamicscited by 12
  37. 2022Is Integer Arithmetic Enough for Deep Learning Training?cited by 11
  38. 2018Non-monotone Behavior of the Heavy Ball Methodcited by 8

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.