Learning with Hidden Factorial Structure
Not on the map. Why?
Abstract (arXiv)
Statistical learning in high-dimensional spaces is challenging without a strong underlying data structure. Recent advances with foundational models suggest that text and image data contain such hidden structures, which help mitigate the curse of dimensionality. Inspired by results from nonparametric statistics, we hypothesize that this phenomenon can be partially explained in terms of decomposition of complex tasks into simpler subtasks. In this paper, we present a controlled experimental framework to test whether neural networks can indeed exploit such "hidden factorial structures". We find that they do leverage these latent patterns to learn discrete distributions more efficiently. We also study the interplay between our structural assumptions and the models' capacity for generalization.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 2 Nov 2024.
Builds on
20 of 33 references.
References · 20
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2016Layer Normalizationcited by 5,579
- 2020GLU Variants Improve Transformercited by 1,724
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
- 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
- 2017Learning Independent Causal Mechanismscited by 104
- 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
- 2020Learning Parities with Neural Networkscited by 77
- 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
- 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 53
8 more
- 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 45
- 2022Causal de Finetti: On the Identification of Invariant Causal Structure in Exchangeable Datacited by 42
- 2022Sub-Task Decomposition Enables Learning in Sequence to Sequence Taskscited by 41
- 2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 35
- 2017Testing Conditional Independence of Discrete Distributionscited by 22
- 2024Memory Mosaicscited by 16
- 2024Attention layers provably solve single-location regressioncited by 15
- 2024SecureLLM: Using Compositionality to Build Provably Secure Language Models for Private, Sensitive, and Secret Datacited by 4
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.