mlsanityData to 14 Oct 2026

Probing the Latent Hierarchical Structure of Data via Diffusion Models

arXiv 2410.13770 ↗First posted 17 Oct 2024stat.ML cond-mat.dis-nn cs.LGcited by 26

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

High-dimensional data must be highly structured to be learnable. Although the compositional and hierarchical nature of data is often put forward to explain learnability, quantitative measurements establishing these properties are scarce. Likewise, accessing the latent variables underlying such a data structure remains a challenge. In this work, we show that forward-backward experiments in diffusion-based models, where data is noised and then denoised to generate new samples, are a promising tool to probe the latent structure of data. We predict in simple hierarchical models that, in this process, changes in data occur by correlated chunks, with a length scale that diverges at a noise level where a phase transition is known to take place. Remarkably, we confirm this prediction in both text and image datasets using state-of-the-art diffusion models. Our results show how latent variable changes manifest in the data and establish how to measure these effects in real data using diffusion models.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 36 of its 57 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

36 of 57 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Models29 Apr
  2. 2024Critical windows: non-asymptotic theory for feature emergence in diffusion models3 Mar
  3. 2024Dynamical Regimes of Diffusion Models28 Feb
  4. 2024A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data26 Feb
  5. 2023The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability26 Oct
  6. 2023U-Turn Diffusion14 Aug
  7. 2023Diffusion Models Generate Images Like Painters: an Analytical Theory of Outline First, Details Later4 Mar
  8. 2022Diffusion Models: A Comprehensive Survey of Methods and Applications2 Sep
  9. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  10. 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul
  11. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  12. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
1 more
  1. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  8. 2022A ConvNet for the 2020scited by 3,579
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  11. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  12. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
11 more
  1. 2022Generative Modelling With Inverse Heat Dissipationcited by 167
  2. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  3. 2017Failures of Gradient-Based Deep Learningcited by 80
  4. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 71
  5. 2024Towards a theory of how the structure of language is acquired by deep neural networkscited by 39
  6. 2016Deep Learning and Hierarchal Generative Modelscited by 25
  7. 2023Learning multi-scale local conditional probability models of imagescited by 21
  8. 2018A Provably Correct Algorithm for Deep Learning that Actually Workscited by 20
  9. 2024How transformers learn structured data: insights from hierarchical filteringcited by 18
  10. 2024Avoiding mode collapse in diffusion models fine-tuned with reinforcement learningcited by 12
  11. 2015A Probabilistic Theory of Deep Learningcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 18.

26 papers.

Sort:

By area:

  1. 2025How Compositional Generalization and Creativity Improve as Diffusion Models are Trainedcited by 25
  2. 2025Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Modelscited by 25
  3. 2025Learning curves theory for hierarchically compositional data with power-law distributed featurescited by 17
  4. 2025A solvable model of learning generative diffusion: theory and insightscited by 14
  5. 2025On the Emergence of Linear Analogies in Word Embeddingscited by 13
  6. 2025The Information Dynamics of Generative Diffusioncited by 13
  7. 2025Local Diffusion Models and Phases of Data Distributionscited by 8
  8. 2026Provable Learning of Random Hierarchy Models and Hierarchical Shallow-to-Deep Chainingcited by 6
  9. 2026How Out-of-Equilibrium Phase Transitions can Seed Pattern Formation in Trained Diffusion Modelscited by 6
  10. 2025Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architecturescited by 5
  11. 2026Biased Generalization in Diffusion Modelscited by 4
  12. 2026Towards Worst-Case Guarantees with Scale-Aware Interpretabilitycited by 3
14 more
  1. 2026Steering Dynamical Regimes of Diffusion Models by Breaking Detailed Balancecited by 3
  2. 2026Concurrence of Symmetry Breaking and Nonlocality Phase Transitions in Diffusion Modelscited by 2
  3. 2026Sampling Data with Chains of Forward-Backward Diffusion Stepscited by 2
  4. 2026Learn from your own latents and not from tokens: A sample-complexity theorycited by 2
  5. 2026Statistical Properties of Training & Generalizationcited by 2
  6. 2026Spectral phase transitions and trainability in neural network learning dynamicscited by 2
  7. 2026Hierarchical Concept Embedding & Pursuit for Interpretable Image Classificationcited by 1
  8. 2026Learning with Shallow Neural Networks on Cluster-Structured Featurescited by 1
  9. 2026Diffusion Models Preferentially Memorize Prototypical Examples or: Why Does My Diffusion Model Love Slop?cited by 1
  10. 2026Parallelism, critical windows, and separations among diffusion language modelscited by 1
  11. 2026Generative models on phase spacecited by 0
  12. 2026DALM: A Domain-Algebraic Language Model via Three-Phase Structured Generationcited by 0
  13. 2026DiffMath: Symbol- and Graph-Aware Latent Diffusion Transformer for Handwritten Mathematical Expression Generationcited by 0
  14. 2026A theory of platonic representations in language modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Latent Structure Emergence in Diffusion Models via Confidence-Based Filtering
  2. 2025Generalization Dynamics of Linear Diffusion Models
  3. 2025Optimal Stopping in Latent Diffusion Models
  4. 2024Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributions
  5. 2026A theory of learning data statistics in diffusion models, from easy to hard
  6. 2024Bigger is not Always Better: Scaling Properties of Latent Diffusion Models
  7. 2025Diffusion models under low-noise regime
  8. 2026First Learn, Then Memorize: The Spectral Bias of Diffusion Models