mlsanityData to 7 Oct 2026

Feature learning from non-Gaussian inputs: the case of Independent Component Analysis in high dimensions

arXiv 2503.23896 ↗First posted 31 Mar 2025stat.ML cond-mat.dis-nn cond-mat.stat-mechcited by 7

Not on the map. Why?

Abstract (arXiv)

Deep neural networks learn structured features from complex, non-Gaussian inputs, but the mechanisms behind this process remain poorly understood. Our work is motivated by the observation that the first-layer filters learnt by deep convolutional neural networks from natural images resemble those learnt by independent component analysis (ICA), a simple unsupervised method that seeks the most non-Gaussian projections of its inputs. This similarity suggests that ICA provides a simple, yet principled model for studying feature learning. Here, we leverage this connection to investigate the interplay between data structure and optimisation in feature learning for the most popular ICA algorithm, FastICA, and stochastic gradient descent (SGD), which is used to train deep networks. We rigorously establish that FastICA requires at least ngtrsim d4 samples to recover a single non-Gaussian direction from d-dimensional inputs on a simple synthetic data model. We show that vanilla online SGD outperforms FastICA, and prove that the optimal sample complexity n gtrsim d2 can be reached by smoothing the loss, albeit in a data-dependent way. We finally demonstrate the existence of a search phase for FastICA on ImageNet, and discuss how the strong non-Gaussianity of said images compensates for the poor sample complexity of FastICA.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 19 of its 45 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

19 of 45 references.

References · 19

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Densely Connected Convolutional Networkscited by 6,691
  3. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  4. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  5. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  6. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  7. 2016Statistical Query Lower Bounds for Robust Estimation of High-dimensional Gaussians and Gaussian Mixturescited by 114
  8. 2019Notes on Computational Hardness of Hypothesis Testing: Predictions using the Low-Degree Likelihood Ratiocited by 82
  9. 2014A statistical model for tensor PCAcited by 72
  10. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  11. 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 60
  12. 2020Statistical Query Algorithms and Low-Degree Tests Are Almost Equivalentcited by 51
7 more
  1. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  2. 2022Data-driven emergence of convolutional structure in neural networkscited by 29
  3. 2022Statistical-Computational Trade-offs in Tensor PCA and Related Problems via Communication Complexitycited by 12
  4. 2023Large Dimensional Independent Component Analysis: Statistical Optimality and Computational Tractabilitycited by 12
  5. 2024Sliding down the stairs: how correlated latent variables accelerate learning with neural networkscited by 12
  6. 2024On the universality of neural encodings in CNNscited by 4
  7. 2025Nonlinear dynamics of localization in neural receptive fieldscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2026A solvable high-dimensional model where nonlinear autoencoders learn structure invisible to PCA while test loss misaligns with generalizationcited by 7
  2. 2026A theory of learning data statistics in diffusion models, from easy to hardcited by 5
  3. 2026Memorisation, convergence and generalisation in generative modelscited by 1
  4. 2025Learning Rate Should Scale Inversely with High-Order Data Moments in High-Dimensional Online Independent Component Analysiscited by 0
  5. 2026Learning Beyond the Gaussian Data: Learning Dynamics of Neural Networks on an Expressive and Cumulant-Controllable Data Modelcited by 0
  6. 2026A 1/R Law for Kurtosis Contrast in Balanced Mixturescited by 0
  7. 2026A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insightscited by 0