# Schedule On the Fly: Diffusion Time Prediction for Faster and Better Image Generation

arXiv: https://arxiv.org/abs/2412.01243 · HTML: https://mlsanity.com/paper/2412.01243/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 40 of its 68 references that are in the map.
- First posted on arXiv 2 Dec 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 40)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [Equivariant Diffusion for Molecule Generation in 3D](https://mlsanity.com/paper/2203.17003/) (2022)
- [DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16381/) (2023)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models](https://mlsanity.com/paper/2305.18455/) (2023)
- [DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation](https://mlsanity.com/paper/2301.03786/) (2023)
- [Align Your Steps: Optimizing Sampling Schedules in Diffusion Models](https://mlsanity.com/paper/2404.14507/) (2024)
- [Diffusion Actor-Critic with Entropy Regulator](https://mlsanity.com/paper/2405.15177/) (2024)
- [Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance](https://mlsanity.com/paper/2111.11755/) (2021)

## Built on by (most cited first, 30 of 30)

- [RewardDance: Reward Scaling in Visual Generation](https://mlsanity.com/paper/2509.08826/) (2025)
- [Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models](https://mlsanity.com/paper/2505.10446/) (2025)
- [Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generation](https://mlsanity.com/paper/2503.13070/) (2025)
- [On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devices](https://mlsanity.com/paper/2502.04363/) (2025)
- [Reinforcing Diffusion Models by Direct Group Preference Optimization](https://mlsanity.com/paper/2510.08425/) (2025)
- [D3P: Dynamic Denoising Diffusion Policy via Reinforcement Learning](https://mlsanity.com/paper/2508.06804/) (2025)
- [Adding Additional Control to One-Step Diffusion with Joint Distribution Matching](https://mlsanity.com/paper/2503.06652/) (2025)
- [fastWDM3D: Fast and Accurate 3D Healthy Tissue Inpainting](https://mlsanity.com/paper/2507.13146/) (2025)
- [TDM-R1: Reinforcing Few-Step Diffusion Models with Non-Differentiable Reward](https://mlsanity.com/paper/2603.07700/) (2026)
- [RAPID^3: Tri-Level Reinforced Acceleration Policies for Diffusion Transformer](https://mlsanity.com/paper/2509.22323/) (2025)
- [SpatialReward: Verifiable Spatial Reward Modeling for Fine-Grained Spatial Consistency in Text-to-Image Generation](https://mlsanity.com/paper/2603.22228/) (2026)
- [TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Models](https://mlsanity.com/paper/2601.08011/) (2026)
- [Just-in-Time: Training-Free Spatial Acceleration for Diffusion Transformers](https://mlsanity.com/paper/2603.10744/) (2026)
- [DiffGraph: An Automated Agent-driven Model Merging Framework for In-the-Wild Text-to-Image Generation](https://mlsanity.com/paper/2603.20470/) (2026)
- [Asynchronous Denoising Diffusion Models for Aligning Text-to-Image Generation](https://mlsanity.com/paper/2510.04504/) (2025)
- [Neural Predictor-Corrector: Solving Homotopy Problems with Reinforcement Learning](https://mlsanity.com/paper/2602.03086/) (2026)
- [Designing Instance-Level Sampling Schedules via REINFORCE with James-Stein Shrinkage](https://mlsanity.com/paper/2511.22177/) (2025)
- [StaTS: Spectral Trajectory Schedule Learning for Adaptive Time Series Forecasting with Frequency Guided Denoiser](https://mlsanity.com/paper/2603.00037/) (2026)
- [Analyzing and Improving Fast Sampling of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2603.00763/) (2026)
- [Learning Sampling Parameters for Diffusion Models](https://mlsanity.com/paper/2607.23488/) (2026)
- [On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devices](https://mlsanity.com/paper/2503.23796/) (2025)
- [Alternators With Noise Models](https://mlsanity.com/paper/2505.12544/) (2025)
- [2ndMatch: Finetuning Pruned Diffusion Models via Second-Order Jacobian Matching](https://mlsanity.com/paper/2506.05398/) (2025)
- [AsyncDiff: Asynchronous Timestep Conditioning for Enhanced Text-to-Image Diffusion Inference](https://mlsanity.com/paper/2512.18675/) (2025)
- [Enforcing Constraints in Generative Sampling via Adaptive Correction Scheduling](https://mlsanity.com/paper/2605.11214/) (2026)
- [Unbiased Diffusion Variational Inversion via Principled Posterior Matching](https://mlsanity.com/paper/2605.25042/) (2026)
- [CSFlow: Aligning Flow Matching with Human Contrast Sensitivity](https://mlsanity.com/paper/2606.08833/) (2026)
- [Quality-Aware Modulation for Diffusion Transformers](https://mlsanity.com/paper/2606.30934/) (2026)
- [Leveraging Inference-Time Compute for Diffusion Models via Global Scheduling of Denoising Trajectories](https://mlsanity.com/paper/2609.22867/) (2026)
- [PixARG: Pixel-Asynchronous Retrieval-Augmented Diffusion for Fast and Realistic Image Generation](https://mlsanity.com/paper/2610.14713/) (2026)

## Abstract (arXiv)

Diffusion and flow matching models have achieved remarkable success in text-to-image generation. However, these models typically rely on the predetermined denoising schedules for all prompts. The multi-step reverse diffusion process can be regarded as a kind of chain-of-thought for generating high-quality images step by step. Therefore, diffusion models should reason for each instance to adaptively determine the optimal noise schedule, achieving high generation quality with sampling efficiency. In this paper, we introduce the Time Prediction Diffusion Model (TPDM) for this. TPDM employs a plug-and-play Time Prediction Module (TPM) that predicts the next noise level based on current latent features at each denoising step. We train the TPM using reinforcement learning to maximize a reward that encourages high final image quality while penalizing excessive denoising steps. With such an adaptive scheduler, TPDM not only generates high-quality images that are aligned closely with human preferences but also adjusts diffusion time and the number of denoising steps on the fly, enhancing both performance and efficiency. With Stable Diffusion 3 Medium architecture, TPDM achieves an aesthetic score of 5.44 and a human preference score (HPS) of 29.59, while using around 50% fewer denoising steps to achieve better performance.