mlsanityData to 7 Oct 2026

Failures of Gradient-Based Deep Learning

arXiv 1703.07950 ↗First posted 23 Mar 2017cs.LG cs.NE stat.MLcited by 79

Not on the map. Why?

Abstract (arXiv)

In recent years, Deep Learning has become the go-to solution for a broad range of applications, often outperforming state-of-the-art. However, it is important, for both theoreticians and practitioners, to gain a deeper understanding of the difficulties and limitations associated with common approaches and algorithms. We describe four types of simple problems, for which the gradient-based algorithms commonly used in deep learning either fail or suffer from significant difficulties. We illustrate the failures through practical experiments, and provide theoretical insights explaining their source, and how they might be remedied.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 12; 2023: 16; 2024: 15; 2025: 14; 2026: 11.

79 papers.

Sort:

By area:

  1. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  2. 2021Quantum machine learning beyond kernel methodscited by 91
  3. 2024A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Datacited by 88
  4. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
  5. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  6. 2023Auto-Regressive Next-Token Predictors are Universal Learnerscited by 65
  7. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  8. 2021Linear algebra with transformerscited by 62
  9. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  10. 2022Sub-Task Decomposition Enables Learning in Sequence to Sequence Taskscited by 41
  11. 2021Recent advances for quantum classifierscited by 35
  12. 2024Iteration Head: A Mechanistic Study of Chain-of-Thoughtcited by 31
38 more
  1. 2022SALSA: Attacking Lattice Cryptography with Transformerscited by 28
  2. 2024From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiencycited by 28
  3. 2021Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programscited by 25
  4. 2024Probing the Latent Hierarchical Structure of Data via Diffusion Modelscited by 25
  5. 2021Techniques for Symbol Grounding with SATNetcited by 24
  6. 2021On the Cryptographic Hardness of Learning Single Periodic Neuronscited by 20
  7. 2022Deep Symbolic Learning: Discovering Symbols and Rules from Perceptionscited by 20
  8. 2021The Connection Between Approximation, Depth Separation and Learnability in Neural Networkscited by 19
  9. 2023A Mathematical Model for Curriculum Learning for Paritiescited by 19
  10. 2021Deep ReLU Networks Preserve Expected Lengthcited by 17
  11. 2022On the non-universality of deep learning: quantifying the cost of symmetrycited by 17
  12. 2021Derivative-Free Reinforcement Learning: A Reviewcited by 15
  13. 2021Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networkscited by 14
  14. 2023Learning Narrow One-Hidden-Layer ReLU Networkscited by 14
  15. 2023Machine learning with tree tensor networks, CP rank constraints, and tensor dropoutcited by 14
  16. 2024On The Specialization of Neural Modulescited by 14
  17. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 14
  18. 2023Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luckcited by 13
  19. 2022DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Predictioncited by 12
  20. 2024Neuro-symbolic Learning Yielding Logical Constraintscited by 12
  21. 2024The Belief State Transformercited by 12
  22. 2022Maximal Initial Learning Rates in Deep ReLU Networkscited by 11
  23. 2025Physics of Skill Learningcited by 8
  24. 2025Hardness of Learning Fixed Parities with Neural Networkscited by 7
  25. 2025Computational Algebra with Attention: Transformer Oracles for Border Basis Algorithmscited by 7
  26. 2023What Makes Data Suitable for a Locally Connected Neural Network? A Necessary and Sufficient Condition Based on Quantum Entanglementcited by 4
  27. 2024Stochastic Reservoir Computerscited by 4
  28. 2024Making Hard Problems Easier with Custom Data Distributions and Loss Regularization: A Case Study in Modular Arithmeticcited by 4
  29. 2022Single Image Super-Resolution Methods: A Surveycited by 3
  30. 2022Impartial Games: A Challenge for Reinforcement Learningcited by 3
  31. 2024Improvement of Spiking Neural Network with Bit Planes and Color Modelscited by 3
  32. 2025Quantum advantage for learning shallow neural networks with natural data distributionscited by 3
  33. 2025LLM Priors for ERM over Programscited by 3
  34. 2023Gradient Descent Fails to Learn High-frequency Functions and Modular Arithmeticcited by 2
  35. 2024Simple and Effective Transfer Learning for Neuro-Symbolic Integrationcited by 2
  36. 2024A Riemannian Optimization Perspective of the Gauss-Newton Method for Feedforward Neural Networkscited by 2
  37. 2025Data Shifts Hurt CoT: A Theoretical Studycited by 2
  38. 2026Near-Universal Multiplicative Updates for Nonnegative Einsum Factorizationcited by 2

50 most cited of 79 All 79 (JSON)