mlsanityData to 7 Oct 2026

How transformers learn structured data: insights from hierarchical filtering

arXiv 2408.15138 ↗First posted 27 Aug 2024cs.LG cond-mat.dis-nn cond-mat.stat-mechcited by 18

Not on the map. Why?

Abstract (arXiv)

Understanding the learning process and the embedded computation in transformers is becoming a central goal for the development of interpretable AI. In the present study, we introduce a hierarchical filtering procedure for data models of sequences on trees, allowing us to hand-tune the range of positional correlations in the data. Leveraging this controlled setting, we provide evidence that vanilla encoder-only transformers can approximate the exact inference algorithm when trained on root classification and masked language modeling tasks, and study how this computation is discovered and implemented. We find that correlations at larger distances, corresponding to increasing layers of the hierarchy, are sequentially included by the network during training. By comparing attention maps from models trained with varying degrees of filtering and by probing the different encoder levels, we find clear evidence of a reconstruction of correlations on successive length scales corresponding to the various levels of the hierarchy, which we relate to a plausible implementation of the exact inference algorithm within the same architecture.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 23 of its 40 references that are in the map. First posted on arXiv 27 Aug 2024.

Builds on

23 of 40 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2021Thinking Like Transformerscited by 182
  7. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  8. 2024A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Datacited by 88
  9. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
  10. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  11. 2023Do Transformers Parse while Predicting the Masked Word?cited by 48
  12. 2024Towards a theory of how the structure of language is acquired by deep neural networkscited by 38
11 more
  1. 2021Curriculum learning for language modelingcited by 32
  2. 2016Deep Learning and Hierarchal Generative Modelscited by 23
  3. 2023Mapping of attention mechanisms to a generalized Potts modelcited by 19
  4. 2013Belief propagation, robust reconstruction and optimal recovery of block modelscited by 18
  5. 2024U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Modelscited by 17
  6. 2024Sliding down the stairs: how correlated latent variables accelerate learning with neural networkscited by 12
  7. 2024How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Modelcited by 7
  8. 2019Emergence of order in random languagescited by 5
  9. 2022Exact Phase Transitions for Stochastic Block Models and Reconstruction on Treescited by 4
  10. 2023Approximating CKY with Transformerscited by 3
  11. 2023Robustness of the Random Language Modelcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 11.

18 papers.

Sort:

By area:

  1. 2024Probing the Latent Hierarchical Structure of Data via Diffusion Modelscited by 25
  2. 2025How Compositional Generalization and Creativity Improve as Diffusion Models are Trainedcited by 24
  3. 2025Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Modelscited by 24
  4. 2024A distributional simplicity bias in the learning dynamics of transformerscited by 23
  5. 2026Deriving Neural Scaling Laws from the statistics of natural languagecited by 18
  6. 2025Learning curves theory for hierarchically compositional data with power-law distributed featurescited by 17
  7. 2025Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architecturescited by 5
  8. 2026A theory of learning data statistics in diffusion models, from easy to hardcited by 5
  9. 2026Biased Generalization in Diffusion Modelscited by 3
  10. 2026Learning with Shallow Neural Networks on Cluster-Structured Featurescited by 1
  11. 2026Learn from your own latents and not from tokens: A sample-complexity theorycited by 1
  12. 2026An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammarscited by 1
6 more
  1. 2025Towards Fundamental Language Models: Does Linguistic Competence Scale with Model Size?cited by 0
  2. 2026Distributional simplicity bias and effective convexity in Energy Based Modelscited by 0
  3. 2026A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attentioncited by 0
  4. 2026A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insightscited by 0
  5. 2026Sampling Data with Chains of Forward-Backward Diffusion Stepscited by 0
  6. 2026Critical Percolation as a Synthetic Data Model for Interpretabilitycited by 0