mlsanityData to 14 Oct 2026

U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Models

arXiv 2404.18444 ↗First posted 29 Apr 2024cs.LG cs.AI math.STcited by 17

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

U-Nets are among the most widely used architectures in computer vision, renowned for their exceptional performance in applications such as image segmentation, denoising, and diffusion modeling. However, a theoretical explanation of the U-Net architecture design has not yet been fully established. This paper introduces a novel interpretation of the U-Net architecture by studying certain generative hierarchical models, which are tree-structured graphical models extensively utilized in both language and image domains. With their encoder-decoder structure, long skip connections, and pooling and up-sampling layers, we demonstrate how U-Nets can naturally implement the belief propagation denoising algorithm in such generative hierarchical models, thereby efficiently approximating the denoising functions. This leads to an efficient sample complexity bound for learning the denoising function using U-Nets within these models. Additionally, we discuss the broader implications of these findings for diffusion models in generative hierarchical models. We also demonstrate that the conventional architecture of convolutional neural networks (ConvNets) is ideally suited for classification tasks within these models. This offers a unified view of the roles of ConvNets and U-Nets, highlighting the versatility of generative hierarchical models in modeling complex data distributions across language and image domains.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 72 of its 106 references that are in the map. First posted on arXiv 29 Apr 2024.

Builds on

72 of 106 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization11 Apr
  2. 2024Unveil Conditional Diffusion Models with Classifier-free Guidance: A Sharp Statistical Theory18 Mar
  3. 2024Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models3 Mar
  4. 2024Dynamical Regimes of Diffusion Models28 Feb
  5. 2024A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data26 Feb
  6. 2023Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Models20 Sep
  7. 2023Nearly d-Linear Convergence Bounds for Diffusion Models via Stochastic Localization7 Aug
  8. 2023Learning Mixtures of Gaussians Using the DDPM Objective3 Jul
  9. 2023Towards Faster Non-Asymptotic Convergence for Diffusion-Based Generative Models15 Jun
  10. 2023A Unified Framework for U-Net Design and Analysis31 May
  11. 2023The probability flow ODE is provably fast19 May
  12. 2023Restoration-Degradation Beyond Linear Diffusions: A Non-Asymptotic Analysis For DDIM-Type Samplers6 Mar
8 more
  1. 2023Diffusion Models are Minimax Optimal Distribution Estimators3 Mar
  2. 2023Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data14 Feb
  3. 2022Improved Analysis of Score-based Generative Modeling: User-Friendly Bounds under Minimal Smoothness Assumptions3 Nov
  4. 2022Convergence of score-based generative modeling for general data distributions26 Sep
  5. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions22 Sep
  6. 2022Let us Build Bridges: Understanding and Extending Diffusion Generative Models31 Aug
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  8. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  3. 2014Going Deeper with Convolutionscited by 4,241
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  5. 2018UNet++: A Nested U-Net Architecture for Medical Image Segmentationcited by 1,525
  6. 2016Variational Inference: A Review for Statisticianscited by 1,472
  7. 2018Attention U-Net: Learning Where to Look for the Pancreascited by 1,435
  8. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  9. 2019Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processingcited by 421
  10. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  11. 2022Transformers Learn Shortcuts to Automatacited by 268
  12. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
38 more
  1. 2023Looped Transformers as Programmable Computerscited by 237
  2. 2015Conditional Random Fields as Recurrent Neural Networkscited by 214
  3. 2021UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformercited by 197
  4. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  5. 2020U-Net and its variants for medical image segmentation: theory and applicationscited by 169
  6. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  7. 2016The Importance of Skip Connections in Biomedical Image Segmentationcited by 126
  8. 2020A Unified View on Graph Neural Networks as Graph Signal Denoisingcited by 110
  9. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 102
  10. 2021Learning with invariances in random features and kernel modelscited by 89
  11. 2018Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholdscited by 85
  12. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  13. 2023White-Box Transformers via Sparse Rate Reductioncited by 80
  14. 2022Improved analysis for a proximal algorithm for samplingcited by 73
  15. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 71
  16. 2016Convolutional Neural Networks Analyzed via Convolutional Sparse Codingcited by 52
  17. 2021A unifying tutorial on Approximate Message Passingcited by 50
  18. 2023Reward-Directed Conditional Diffusion: Provable Distribution Estimation and Reward Improvementcited by 49
  19. 2022Sampling from the Sherrington-Kirkpatrick Gibbs measure via algorithmic stochastic localizationcited by 48
  20. 2023Sampling, Diffusions, and Stochastic Localizationcited by 40
  21. 2023Analysis of learning a flow-based generative model from limited sample complexitycited by 38
  22. 2020Why Are Convolutional Nets More Sample-Efficient than Fully-Connected Nets?cited by 34
  23. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
  24. 2021Locality defeats the curse of dimensionality in convolutional teacher-student scenarioscited by 26
  25. 2023Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspectivecited by 26
  26. 2016Deep Learning and Hierarchal Generative Modelscited by 25
  27. 2023Emergence of Segmentation with Minimalistic White-Box Transformerscited by 24
  28. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  29. 2023Theoretical Analysis of Inductive Biases in Deep Convolutional Networkscited by 21
  30. 2021Parametric Complexity Bounds for Approximating PDEs with Neural Networkscited by 19
  31. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  32. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  33. 2021Local convexity of the TAP free energy and AMP convergence for Z2-synchronizationcited by 15
  34. 2022Neural Network Approximations of PDEs Beyond Linearity: A Representational Perspectivecited by 15
  35. 2018An Instability in Variational Inference for Topic Modelscited by 11
  36. 2022Sudakov-Fernique post-AMP, and a new proof of the local convexity of the TAP free energycited by 9
  37. 2023A Multi-Resolution Framework for U-Nets with Applications to Hierarchical VAEscited by 9
  38. 2024How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Modelcited by 7

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 9; 2026: 3.

17 papers.

Sort:

By area:

  1. 2024Towards a theory of how the structure of language is acquired by deep neural networkscited by 39
  2. 2024Probing the Latent Hierarchical Structure of Data via Diffusion Modelscited by 26
  3. 2025How Compositional Generalization and Creativity Improve as Diffusion Models are Trainedcited by 25
  4. 2025Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Modelscited by 25
  5. 2024How transformers learn structured data: insights from hierarchical filteringcited by 18
  6. 2025Learning curves theory for hierarchically compositional data with power-law distributed featurescited by 17
  7. 2025A solvable model of learning generative diffusion: theory and insightscited by 14
  8. 2026Provable Learning of Random Hierarchy Models and Hierarchical Shallow-to-Deep Chainingcited by 6
  9. 2024Diffusion Transformer Captures Spatial-Temporal Dependencies: A Theory for Gaussian Process Datacited by 5
  10. 2025Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architecturescited by 5
  11. 2026Deep networks learn to parse uniform-depth context-free languages from local statisticscited by 5
  12. 2025Transformers as Unsupervised Learning Algorithms: A study on Gaussian Mixturescited by 3
5 more
  1. 2025Time-Embedded Algorithm Unrolling for Computational MRIcited by 2
  2. 2026Learn from your own latents and not from tokens: A sample-complexity theorycited by 2
  3. 2024Unrolled denoising networks provably learn optimal Bayesian inferencecited by 1
  4. 2025Dynamic Importance in Diffusion U-Net for Enhanced Image Synthesiscited by 1
  5. 2025CHEM: Estimating and Understanding Hallucinations in Deep Learning for Image Processingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Diffusion Models Beat GANs on Image Classification
  2. 2026UDT: Reconciling U-Nets and Diffusion Transformers with Data-Adaptive Token Reduction
  3. 2023FreeU: Free Lunch in Diffusion U-Net
  4. 2024Neural Residual Diffusion Models for Deep Scalable Vision Generation
  5. 2023Factorized Diffusion Architectures for Unsupervised Image Generation and Segmentation
  6. 2024U-KAN Makes Strong Backbone for Medical Image Segmentation and Generation
  7. 2023Conditional Generation from Unconditional Diffusion Models using Denoiser Representations
  8. 2022All are Worth Words: A ViT Backbone for Diffusion Models