mlsanityData to 7 Oct 2026

Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm

arXiv 2207.04612 ↗First posted 11 Jul 2022cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Although learning in high dimensions is commonly believed to suffer from the curse of dimensionality, modern machine learning methods often exhibit an astonishing power to tackle a wide range of challenging real-world learning problems without using abundant amounts of data. How exactly these methods break this curse remains a fundamental open question in the theory of deep learning. While previous efforts have investigated this question by studying the data (D), model (M), and inference algorithm (I) as independent modules, in this paper, we analyze the triplet (D, M, I) as an integrated system and identify important synergies that help mitigate the curse of dimensionality. We first study the basic symmetries associated with various learning algorithms (M, I), focusing on four prototypical architectures in deep learning: fully-connected networks (FCN), locally-connected networks (LCN), and convolutional networks with and without pooling (GAP/VEC). We find that learning is most efficient when these symmetries are compatible with those of the data distribution and that performance significantly deteriorates when any member of the (D, M, I) triplet is inconsistent or suboptimal.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 59 of its 86 references that are in the map. First posted on arXiv 11 Jul 2022.

Builds on

59 of 86 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  9. 2018On the Spectral Bias of Neural Networkscited by 1,321
  10. 2016Group Equivariant Convolutional Networkscited by 1,154
  11. 2021Scaling Vision Transformerscited by 993
  12. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
38 more
  1. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  2. 2018On Lazy Training in Differentiable Programmingcited by 671
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  4. 2017Deep Neural Networks as Gaussian Processescited by 617
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  6. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  7. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  8. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  9. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  10. 2021Explaining Neural Scaling Lawscited by 353
  11. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  12. 2017Implicit Regularization in Matrix Factorizationcited by 318
  13. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  14. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  15. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 307
  16. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  17. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  18. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  19. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  20. 2016Deep Information Propagationcited by 194
  21. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  22. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  23. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  24. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  25. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  26. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  27. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  28. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  29. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  30. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  31. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  32. 2017Implicit Regularization in Deep Learningcited by 90
  33. 2021Learning with invariances in random features and kernel modelscited by 89
  34. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  35. 2020The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networkscited by 58
  36. 2017SGD Learns the Conjugate Kernel Class of the Networkcited by 54
  37. 2020Towards Learning Convolutions from Scratchcited by 46
  38. 2018Understanding training and generalization in deep learning by Fourier analysiscited by 43

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 1; 2024: 2.

6 papers.

  1. 2023Theoretical Analysis of Inductive Biases in Deep Convolutional Networkscited by 21
  2. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  3. 2022The Curious Case of Benign Memorizationcited by 11
  4. 2024Which Frequencies do CNNs Need? Emergent Bottleneck Structure in Feature Learningcited by 9
  5. 2024How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Modelcited by 7
  6. 2022An open unified deep graph learning framework for discovering drug leadscited by 0