mlsanityData to 7 Oct 2026

Data-driven emergence of convolutional structure in neural networks

arXiv 2202.00565 ↗First posted 1 Feb 2022cond-mat.dis-nn q-bio.NC stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

Exploiting data invariances is crucial for efficient learning in both artificial and biological neural circuits. Understanding how neural networks can discover appropriate representations capable of harnessing the underlying symmetries of their inputs is thus crucial in machine learning and neuroscience. Convolutional neural networks, for example, were designed to exploit translation symmetry and their capabilities triggered the first wave of deep learning successes. However, learning convolutions directly from translation-invariant data with a fully-connected network has so far proven elusive. Here, we show how initially fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs, resulting in localised, space-tiling receptive fields. These receptive fields match the filters of a convolutional network trained on the same task. By carefully designing data models for the visual scene, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs, which has long been recognised as the hallmark of natural images. We provide an analytical and numerical characterisation of the pattern-formation mechanism responsible for this phenomenon in a simple model and find an unexpected link between receptive field formation and tensor decomposition of higher-order input correlations. These results provide a new perspective on the development of low-level feature detectors in various sensory modalities, and pave the way for studying the impact of higher-order statistics on learning in neural networks.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 35 of its 111 references that are in the map. First posted on arXiv 1 Feb 2022.

Builds on

35 of 111 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2018On the Spectral Bias of Neural Networkscited by 1,321
  4. 2016Group Equivariant Convolutional Networkscited by 1,154
  5. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  6. 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
  7. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  8. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  9. 2016Geometric deep learning on graphs and manifolds using mixture model CNNscited by 468
  10. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  11. 20183D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Datacited by 348
  12. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
23 more
  1. 2019Gauge Equivariant Convolutional Networks and the Icosahedral CNNcited by 262
  2. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  3. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  4. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  5. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  6. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  7. 2016TensorLy: Tensor Learning in Pythoncited by 104
  8. 2021Learning with invariances in random features and kernel modelscited by 89
  9. 2016Understanding Deep Convolutional Networkscited by 76
  10. 2014A statistical model for tensor PCAcited by 72
  11. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  12. 2020Towards Learning Convolutions from Scratchcited by 46
  13. 2019Equivariant Hamiltonian Flowscited by 39
  14. 2016Statistical limits of spiked tensor modelscited by 27
  15. 2018On the Spectrum of Random Features Maps of High Dimensional Datacited by 27
  16. 2021Locality defeats the curse of dimensionality in convolutional teacher-student scenarioscited by 26
  17. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  18. 2019Finding the Needle in the Haystack with Convolutions: on the benefits of architectural biascited by 19
  19. 2020Going in circles is the way forward: the role of recurrence in visual inferencecited by 19
  20. 2021A Random Matrix Perspective on Random Tensorscited by 12
  21. 2019'Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight spacecited by 10
  22. 2021Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones?cited by 3
  23. 2021Tensor decomposition of higher-order correlations by nonlinear Hebbian plasticitycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 8; 2024: 5; 2025: 2; 2026: 7.

29 papers.

Sort:

By area:

  1. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
  2. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  3. 2022Gaussian Universality of Perceptrons with Random Labelscited by 26
  4. 2023Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimationcited by 25
  5. 2022Learning sparse features can lead to overfitting in neural networkscited by 23
  6. 2023Mapping of attention mechanisms to a generalized Potts modelcited by 19
  7. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  8. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  9. 2024Sliding down the stairs: how correlated latent variables accelerate learning with neural networkscited by 12
  10. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  11. 2025Feature learning from non-Gaussian inputs: the case of Independent Component Analysis in high dimensionscited by 7
  12. 2022Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithmcited by 6
17 more
  1. 2023Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisationcited by 6
  2. 2026A theory of learning data statistics in diffusion models, from easy to hardcited by 5
  3. 2023Learning from higher-order statistics, efficiently: hypothesis tests, random features, and neural networkscited by 3
  4. 2022The impact of memory on learning sequence-to-sequence taskscited by 1
  5. 2022A simple probabilistic neural network for machine understandingcited by 1
  6. 2024On How Iterative Magnitude Pruning Discovers Local Receptive Fields in Fully Connected Neural Networkscited by 1
  7. 2025Nonlinear dynamics of localization in neural receptive fieldscited by 1
  8. 2023Optimal transfer protocol by incremental layer defrostingcited by 0
  9. 2023Loss Dynamics of Temporal Difference Reinforcement Learningcited by 0
  10. 2024Classifying Overlapping Gaussian Mixtures in High Dimensions: From Optimal Classifiers to Neural Netscited by 0
  11. 2024Absolute abstraction: a renormalisation group approachcited by 0
  12. 2026Transformers self-organize like newborn visual systems when trained in prenatal worldscited by 0
  13. 2026Learning Beyond the Gaussian Data: Learning Dynamics of Neural Networks on an Expressive and Cumulant-Controllable Data Modelcited by 0
  14. 2026A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insightscited by 0
  15. 2026Cognitive Convergence: Deep Similarities Between Large Language Models and Human Cognitioncited by 0
  16. 2026Coarse-Graining Hidden Representations: Unsupervised Neuron Selection via Mapping Entropycited by 0
  17. 2026The Computational Value of Sensory-Aligned Receptive Fields Depends on Neuronal Expressivitycited by 0