# Inf-DiT: Upsampling Any-Resolution Image with Memory-Efficient Diffusion Transformer

arXiv: https://arxiv.org/abs/2405.04312 · HTML: https://mlsanity.com/paper/2405.04312/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models.
- Builds on 30 of its 33 references that are in the map.
- First posted on arXiv 7 May 2024.

## Placed in

- Diffusion models / [Inverse problems and image restoration](https://mlsanity.com/diffusion/inverse-problems-and-image-restoration/)

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data](https://mlsanity.com/paper/2107.10833/) (2021)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [Designing a Practical Degradation Model for Deep Blind Image Super-Resolution](https://mlsanity.com/paper/2103.14006/) (2021)
- [Exploiting Diffusion Prior for Real-World Image Super-Resolution](https://mlsanity.com/paper/2305.07015/) (2023)
- [DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior](https://mlsanity.com/paper/2308.15070/) (2023)
- [Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylization](https://mlsanity.com/paper/2308.14469/) (2023)
- [ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models](https://mlsanity.com/paper/2310.07702/) (2023)
- [DemoFusion: Democratising High-Resolution Image Generation With No ](https://mlsanity.com/paper/2311.16973/) (2023)
- [Any-resolution Training for High-resolution Image Synthesis](https://mlsanity.com/paper/2204.07156/) (2022)
- [Mixture of Diffusers for scene composition and high resolution image generation](https://mlsanity.com/paper/2302.02412/) (2023)

## Built on by (most cited first, 14 of 14)

- [SeedVR: Seeding Infinity in Diffusion Transformer Towards Generic Video Restoration](https://mlsanity.com/paper/2501.01320/) (2025)
- [Is One GPU Enough? Pushing Image Generation at Higher-Resolutions with Foundation Models](https://mlsanity.com/paper/2406.07251/) (2024)
- [Audio Super-Resolution with Latent Bridge Models](https://mlsanity.com/paper/2509.17609/) (2025)
- [Diffusion Models Need Visual Priors for Image Generation](https://mlsanity.com/paper/2410.08531/) (2024)
- [HiCache: A Plug-in Scaled-Hermite Upgrade for Taylor-Style Cache-then-Forecast Diffusion Acceleration](https://mlsanity.com/paper/2508.16984/) (2025)
- [InfGen: A Resolution-Agnostic Paradigm for Scalable Image Synthesis](https://mlsanity.com/paper/2509.10441/) (2025)
- [LR0.FM: Low-Res Benchmark and Improving Robustness for Zero-Shot Classification in Foundation Models](https://mlsanity.com/paper/2502.03950/) (2025)
- [M3amba: CLIP-driven Mamba Model for Multi-modal Remote Sensing Classification](https://mlsanity.com/paper/2503.06446/) (2025)
- [QDM: Quadtree-Based Region-Adaptive Sparse Diffusion Models for Efficient Image Super-Resolution](https://mlsanity.com/paper/2503.12015/) (2025)
- [CASR: A Robust Cyclic Framework for Arbitrary Large-Scale Super-Resolution with Distribution Alignment and Self-Similarity Awareness](https://mlsanity.com/paper/2602.22159/) (2026)
- [HaineiFRDM: Structure-Preserving Diffusion for Film Restoration under Fast Motion and Diverse Defects](https://mlsanity.com/paper/2512.24946/) (2025)
- [RPT-SR: Regional Prior attention Transformer for infrared image Super-Resolution](https://mlsanity.com/paper/2602.15490/) (2026)
- [Optimizing ID Consistency in Multimodal Large Models: Facial Restoration via Alignment, Entanglement, and Disentanglement](https://mlsanity.com/paper/2602.18752/) (2026)
- [Tuning-Free Latent Diffusion Models for Ultrahigh-Resolution Image Editing](https://mlsanity.com/paper/2607.06136/) (2026)

## Abstract (arXiv)

Diffusion models have shown remarkable performance in image generation in recent years. However, due to a quadratic increase in memory during generating ultra-high-resolution images (e.g. 4096*4096), the resolution of generated images is often limited to 1024*1024. In this work. we propose a unidirectional block attention mechanism that can adaptively adjust the memory overhead during the inference process and handle global dependencies. Building on this module, we adopt the DiT structure for upsampling and develop an infinite super-resolution model capable of upsampling images of various shapes and resolutions. Comprehensive experiments show that our model achieves SOTA performance in generating ultra-high-resolution images in both machine and human evaluation. Compared to commonly used UNet structures, our model can save more than 5x memory when generating 4096*4096 images. The project URL is https://github.com/THUDM/Inf-DiT.