# Restart Sampling for Improving Generative Processes

arXiv: https://arxiv.org/abs/2306.14878 · HTML: https://mlsanity.com/paper/2306.14878/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 26 Jun 2023.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 17 of 17)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [Gotta Go Fast When Generating Data with Score-Based Models](https://mlsanity.com/paper/2105.14080/) (2021)
- [User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient](https://mlsanity.com/paper/1710.00095/) (2017)
- [Poisson Flow Generative Models](https://mlsanity.com/paper/2209.11178/) (2022)
- [PFGM++: Unlocking the Potential of Physics-Inspired Generative Models](https://mlsanity.com/paper/2302.04265/) (2023)

## Built on by (most cited first, 30 of 84)

- [Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion](https://mlsanity.com/paper/2310.02279/) (2023)
- [Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design](https://mlsanity.com/paper/2402.04997/) (2024)
- [Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer](https://mlsanity.com/paper/2311.17009/) (2023)
- [Dirichlet Flow Matching with Applications to DNA Sequence Design](https://mlsanity.com/paper/2402.05841/) (2024)
- [Inductive Moment Matching](https://mlsanity.com/paper/2503.07565/) (2025)
- [Elucidating the Exposure Bias in Diffusion Models](https://mlsanity.com/paper/2308.15321/) (2023)
- [Align Your Steps: Optimizing Sampling Schedules in Diffusion Models](https://mlsanity.com/paper/2404.14507/) (2024)
- [Particle Guidance: non-I.I.D. Diverse Sampling with Diffusion Models](https://mlsanity.com/paper/2310.13102/) (2023)
- [A Survey on Diffusion Models for Time Series and Spatio-Temporal Data](https://mlsanity.com/paper/2404.18886/) (2024)
- [TabDiff: a Mixed-type Diffusion Model for Tabular Data Generation](https://mlsanity.com/paper/2410.20626/) (2024)
- [StableMoFusion: Towards Robust and Efficient Diffusion-based Motion Generation Framework](https://mlsanity.com/paper/2405.05691/) (2024)
- [Inference-time Scaling of Diffusion Models through Classical Search](https://mlsanity.com/paper/2505.23614/) (2025)
- [Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms](https://mlsanity.com/paper/2502.00234/) (2025)
- [Trajectory Consistency Distillation: Improved Latent Consistency Distillation by Semi-Linear Consistency Function with Trajectory Mapping](https://mlsanity.com/paper/2402.19159/) (2024)
- [One-step Diffusion Models with f-Divergence Distribution Matching](https://mlsanity.com/paper/2502.15681/) (2025)
- [Accelerating Diffusion Models with Parallel Sampling: Inference at Sub-Linear Time Complexity](https://mlsanity.com/paper/2405.15986/) (2024)
- [textitJump Your Steps: Optimizing Sampling Schedule of Discrete Diffusion Models](https://mlsanity.com/paper/2410.07761/) (2024)
- [Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcing](https://mlsanity.com/paper/2503.19385/) (2025)
- [Efficient Diffusion Models: A Survey](https://mlsanity.com/paper/2502.06805/) (2025)
- [Spatiotemporal Skip Guidance for Enhanced Video Diffusion Sampling](https://mlsanity.com/paper/2411.18664/) (2024)
- [Understanding and Improving Training-free Loss-based Diffusion Guidance](https://mlsanity.com/paper/2403.12404/) (2024)
- [Object-aware Inversion and Reassembly for Image Editing](https://mlsanity.com/paper/2310.12149/) (2023)
- [Test-Time Scaling of Diffusion Models via Noise Trajectory Search](https://mlsanity.com/paper/2506.03164/) (2025)
- [La-Proteina: Atomistic Protein Generation via Partially Latent Flow Matching](https://mlsanity.com/paper/2507.09466/) (2025)
- [CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation](https://mlsanity.com/paper/2310.01407/) (2023)
- [Truncated Consistency Models](https://mlsanity.com/paper/2410.14895/) (2024)
- [FRDiff : Feature Reuse for Universal Training-free Acceleration of Diffusion Models](https://mlsanity.com/paper/2312.03517/) (2023)
- [DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents](https://mlsanity.com/paper/2407.03300/) (2024)
- [Taming Video Models for 3D and 4D Generation via Zero-Shot Camera Control](https://mlsanity.com/paper/2509.15130/) (2025)
- [Not-So-Optimal Transport Flows for 3D Point Cloud Generation](https://mlsanity.com/paper/2502.12456/) (2025)

## Abstract (arXiv)

Generative processes that involve solving differential equations, such as diffusion models, frequently necessitate balancing speed and quality. ODE-based samplers are fast but plateau in performance while SDE-based samplers deliver higher sample quality at the cost of increased sampling time. We attribute this difference to sampling errors: ODE-samplers involve smaller discretization errors while stochasticity in SDE contracts accumulated errors. Based on these findings, we propose a novel sampling algorithm called Restart in order to better balance discretization errors and contraction. The sampling method alternates between adding substantial noise in additional forward steps and strictly following a backward ODE. Empirically, Restart sampler surpasses previous SDE and ODE samplers in both speed and accuracy. Restart not only outperforms the previous best SDE results, but also accelerates the sampling speed by 10-fold / 2-fold on CIFAR-10 / ImageNet 64 × 64. In addition, it attains significantly better sample quality than ODE samplers within comparable sampling times. Moreover, Restart better balances text-image alignment/visual quality versus diversity than previous samplers in the large-scale text-to-image Stable Diffusion model pre-trained on LAION 512 × 512. Code is available at https://github.com/Newbeeer/diffusionrestartsampling