# Make a Cheap Scaling: A Self-Cascade Diffusion Model for Higher-Resolution Adaptation

arXiv: https://arxiv.org/abs/2402.10491 · HTML: https://mlsanity.com/paper/2402.10491/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 34 of its 37 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Palette: Image-to-Image Diffusion Models](https://mlsanity.com/paper/2111.05826/) (2021)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [Simple diffusion: End-to-end diffusion for high resolution images](https://mlsanity.com/paper/2301.11093/) (2023)
- [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023)
- [Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2309.15818/) (2023)
- [Dual Diffusion Implicit Bridges for Image-to-Image Translation](https://mlsanity.com/paper/2203.08382/) (2022)
- [FreeU: Free Lunch in Diffusion U-Net](https://mlsanity.com/paper/2309.11497/) (2023)
- [On the Importance of Noise Scheduling for Diffusion Models](https://mlsanity.com/paper/2301.10972/) (2023)
- [Video Probabilistic Diffusion Models in Projected Latent Space](https://mlsanity.com/paper/2302.07685/) (2023)
- [ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models](https://mlsanity.com/paper/2310.07702/) (2023)
- [DemoFusion: Democratising High-Resolution Image Generation With No ](https://mlsanity.com/paper/2311.16973/) (2023)
- [DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2304.06648/) (2023)
- [Relay Diffusion: Unifying diffusion process across resolutions for image synthesis](https://mlsanity.com/paper/2309.03350/) (2023)
- [Any-resolution Training for High-resolution Image Synthesis](https://mlsanity.com/paper/2204.07156/) (2022)
- [Matryoshka Diffusion Models](https://mlsanity.com/paper/2310.15111/) (2023)

## Built on by (most cited first, 30 of 51)

- [Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers](https://mlsanity.com/paper/2405.05945/) (2024)
- [VideoDPO: Omni-Preference Alignment for Video Diffusion Generation](https://mlsanity.com/paper/2412.14167/) (2024)
- [Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesis](https://mlsanity.com/paper/2410.08261/) (2024)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [DiffuseHigh: Training-free Progressive High-Resolution Image Synthesis through Structure Guidance](https://mlsanity.com/paper/2406.18459/) (2024)
- [Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget](https://mlsanity.com/paper/2407.15811/) (2024)
- [FreeScale: Unleashing the Resolution of Diffusion Models via Tuning-Free Scale Fusion](https://mlsanity.com/paper/2412.09626/) (2024)
- [MegaFusion: Extend Diffusion Models towards Higher-resolution Image Generation without Further Tuning](https://mlsanity.com/paper/2408.11001/) (2024)
- [Latent Space Super-Resolution for Higher-Resolution Image Generation with Diffusion Models](https://mlsanity.com/paper/2503.18446/) (2025)
- [DynamicScaler: Seamless and Scalable Video Generation for Panoramic Scenes](https://mlsanity.com/paper/2412.11100/) (2024)
- [HiFlow: Training-free High-Resolution Image Generation with Flow-Aligned Guidance](https://mlsanity.com/paper/2504.06232/) (2025)
- [HiPrompt: Tuning-free Higher-Resolution Generation with Hierarchical MLLM Prompts](https://mlsanity.com/paper/2409.02919/) (2024)
- [PhysCorr: Dual-Reward DPO for Physics-Constrained Text-to-Video Generation with Automated Preference Selection](https://mlsanity.com/paper/2511.03997/) (2025)
- [LUVE : Latent-Cascaded Ultra-High-Resolution Video Generation with Dual Frequency Experts](https://mlsanity.com/paper/2602.11564/) (2026)
- [Is One GPU Enough? Pushing Image Generation at Higher-Resolutions with Foundation Models](https://mlsanity.com/paper/2406.07251/) (2024)
- [CutDiffusion: A Simple, Fast, Cheap, and Strong Diffusion Extrapolation Method](https://mlsanity.com/paper/2404.15141/) (2024)
- [Turbo2K: Towards Ultra-Efficient and High-Quality 2K Video Synthesis](https://mlsanity.com/paper/2504.14470/) (2025)
- [UltraImage: Rethinking Resolution Extrapolation in Image Diffusion Transformers](https://mlsanity.com/paper/2512.04504/) (2025)
- [DiVE: Efficient Multi-View Driving Scenes Generation Based on Video Diffusion Transformer](https://mlsanity.com/paper/2504.19614/) (2025)
- [Chain-of-Zoom: Extreme Super-Resolution via Scale Autoregression and Preference Alignment](https://mlsanity.com/paper/2505.18600/) (2025)
- [HiStream: Efficient High-Resolution Video Generation via Redundancy-Eliminated Streaming](https://mlsanity.com/paper/2512.21338/) (2025)
- [Latent Wavelet Diffusion For Ultra-High-Resolution Image Synthesis](https://mlsanity.com/paper/2506.00433/) (2025)
- [RepLDM: Reprogramming Pretrained Latent Diffusion Models for High-Quality, High-Efficiency, High-Resolution Image Generation](https://mlsanity.com/paper/2410.06055/) (2024)
- [ScaleDiff: Higher-Resolution Image Synthesis via Efficient and Model-Agnostic Diffusion](https://mlsanity.com/paper/2510.25818/) (2025)
- [UltraFlux: Data-Model Co-Design for High-quality Native 4K Text-to-Image Generation across Diverse Aspect Ratios](https://mlsanity.com/paper/2511.18050/) (2025)
- [FLAME Diffuser: Wildfire Image Synthesis using Mask Guided Diffusion](https://mlsanity.com/paper/2403.03463/) (2024)
- [PatchVSR: Breaking Video Diffusion Resolution Limits with Patch-wise Video Super-Resolution](https://mlsanity.com/paper/2509.26025/) (2025)
- [AccDiffusion v2: Towards More Accurate Higher-Resolution Diffusion Extrapolation](https://mlsanity.com/paper/2412.02099/) (2024)
- [LEDiT: Your Length-Extrapolatable Diffusion Transformer without Positional Encoding](https://mlsanity.com/paper/2503.04344/) (2025)
- [Exploring Position Encoding in Diffusion U-Net for Training-free High-resolution Image Generation](https://mlsanity.com/paper/2503.09830/) (2025)

## Abstract (arXiv)

Diffusion models have proven to be highly effective in image and video generation; however, they encounter challenges in the correct composition of objects when generating images of varying sizes due to single-scale training data. Adapting large pre-trained diffusion models to higher resolution demands substantial computational and optimization resources, yet achieving generation capabilities comparable to low-resolution models remains challenging. This paper proposes a novel self-cascade diffusion model that leverages the knowledge gained from a well-trained low-resolution image/video generation model, enabling rapid adaptation to higher-resolution generation. Building on this, we employ the pivot replacement strategy to facilitate a tuning-free version by progressively leveraging reliable semantic guidance derived from the low-resolution model. We further propose to integrate a sequence of learnable multi-scale upsampler modules for a tuning version capable of efficiently learning structural details at a new scale from a small amount of newly acquired high-resolution training data. Compared to full fine-tuning, our approach achieves a 5× training speed-up and requires only 0.002M tuning parameters. Extensive experiments demonstrate that our approach can quickly adapt to higher-resolution image and video synthesis by fine-tuning for just 10k steps, with virtually no additional inference time.