# Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformers

arXiv: https://arxiv.org/abs/2401.11605 · HTML: https://mlsanity.com/paper/2401.11605/

## Key facts

- Cited by 118 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 53 of its 69 references that are in the map.
- First posted on arXiv 21 Jan 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)

## Built on by (most cited first, 30 of 118)

- [Pyramidal Flow Matching for Efficient Video Generative Modeling](https://mlsanity.com/paper/2410.05954/) (2024)
- [Guiding a Diffusion Model with a Bad Version of Itself](https://mlsanity.com/paper/2406.02507/) (2024)
- [JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation](https://mlsanity.com/paper/2411.07975/) (2024)
- [Consistency Models Made Easy](https://mlsanity.com/paper/2406.14548/) (2024)
- [ZigMa: A DiT-style Zigzag Mamba Diffusion Model](https://mlsanity.com/paper/2403.13802/) (2024)
- [U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformers](https://mlsanity.com/paper/2405.02730/) (2024)
- [Community Forensics: Using Thousands of Generators to Train Fake Image Detectors](https://mlsanity.com/paper/2411.04125/) (2024)
- [TEXGen: a Generative Diffusion Model for Mesh Textures](https://mlsanity.com/paper/2411.14740/) (2024)
- [Improved Noise Schedule for Diffusion Training](https://mlsanity.com/paper/2407.03297/) (2024)
- [Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization](https://mlsanity.com/paper/2503.11056/) (2025)
- [Posterior-Mean Rectified Flow: Towards Minimum MSE Photo-Realistic Image Restoration](https://mlsanity.com/paper/2410.00418/) (2024)
- [U-REPA: Aligning Diffusion U-Nets to ViTs](https://mlsanity.com/paper/2503.18414/) (2025)
- [SlimFlow: Training Smaller One-Step Diffusion Models with Rectified Flow](https://mlsanity.com/paper/2407.12718/) (2024)
- [Hierarchical Patch Diffusion Models for High-Resolution Video Generation](https://mlsanity.com/paper/2406.07792/) (2024)
- [Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediators](https://mlsanity.com/paper/2408.05710/) (2024)
- [LaVin-DiT: Large Vision Diffusion Transformer](https://mlsanity.com/paper/2411.11505/) (2024)
- [Alfie: Democratising RGBA Image Generation With No ](https://mlsanity.com/paper/2408.14826/) (2024)
- [Generalized Neighborhood Attention: Multi-dimensional Sparse Attention at the Speed of Light](https://mlsanity.com/paper/2504.16922/) (2025)
- [DiffLocks: Generating 3D Hair from a Single Image using Diffusion Models](https://mlsanity.com/paper/2505.06166/) (2025)
- [Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer](https://mlsanity.com/paper/2509.24899/) (2025)
- [Fast LiDAR Data Generation with Rectified Flows](https://mlsanity.com/paper/2412.02241/) (2024)
- [DiC: Rethinking Conv3x3 Designs in Diffusion Models](https://mlsanity.com/paper/2501.00603/) (2024)
- [Second FRCSyn-onGoing: Winning Solutions and Post-Challenge Analysis to Improve Face Recognition with Synthetic Data](https://mlsanity.com/paper/2412.01383/) (2024)
- [Adapting Self-Supervised Representations as a Latent Space for Efficient Generation](https://mlsanity.com/paper/2510.14630/) (2025)
- [Effective Cloud Removal for Remote Sensing Images by an Improved Mean-Reverting Denoising Model with Elucidated Design Space](https://mlsanity.com/paper/2503.23717/) (2025)
- [InvFusion: Bridging Supervised and Zero-shot Diffusion for Inverse Problems](https://mlsanity.com/paper/2504.01689/) (2025)
- [Scaling Up AI-Generated Image Detection with Generator-Aware Prototypes](https://mlsanity.com/paper/2512.12982/) (2025)
- [Phy-Diff: Physics-guided Hourglass Diffusion Model for Diffusion MRI Synthesis](https://mlsanity.com/paper/2406.03002/) (2024)
- [VariFace: Fair and Diverse Synthetic Dataset Generation for Face Recognition](https://mlsanity.com/paper/2412.06235/) (2024)
- [MirrorVerse: Pushing Diffusion Models to Realistically Reflect the World](https://mlsanity.com/paper/2504.15397/) (2025)

## Abstract (arXiv)

We present the Hourglass Diffusion Transformer (HDiT), an image generative model that exhibits linear scaling with pixel count, supporting training at high-resolution (e.g. 1024 × 1024) directly in pixel-space. Building on the Transformer architecture, which is known to scale to billions of parameters, it bridges the gap between the efficiency of convolutional U-Nets and the scalability of Transformers. HDiT trains successfully without typical high-resolution training techniques such as multiscale architectures, latent autoencoders or self-conditioning. We demonstrate that HDiT performs competitively with existing models on ImageNet 2562, and sets a new state-of-the-art for diffusion models on FFHQ-10242.