# SpecDiff: Accelerating Diffusion Model Inference with Self-Speculation

arXiv: https://arxiv.org/abs/2509.13848 · HTML: https://mlsanity.com/paper/2509.13848/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 17 Sep 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024)
- [Text-to-image Diffusion Models in Generative AI: A Survey](https://mlsanity.com/paper/2303.07909/) (2023)
- [GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium](https://mlsanity.com/paper/1706.08500/) (2017)
- [Accelerating Diffusion Transformers with Token-wise Feature Caching](https://mlsanity.com/paper/2410.05317/) (2024)
- [FORA: Fast-Forward Caching in Diffusion Transformer Acceleration](https://mlsanity.com/paper/2407.01425/) (2024)
- [From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeers](https://mlsanity.com/paper/2503.06923/) (2025)
- [Stability](https://mlsanity.com/paper/1310.0150/) (2013)
- [Region-Adaptive Sampling for Diffusion Transformers](https://mlsanity.com/paper/2502.10389/) (2025)
- [SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting](https://mlsanity.com/paper/2504.08850/) (2025)
- [QuantCache: Adaptive Importance-Guided Quantization with Hierarchical Latent and Layer Caching for Video Generation](https://mlsanity.com/paper/2503.06545/) (2025)
- [AdaDiff: Adaptive Step Selection for Fast Diffusion Models](https://mlsanity.com/paper/2311.14768/) (2023)
- [VGDFR: Diffusion-based Video Generation with Dynamic Latent Frame Rate](https://mlsanity.com/paper/2504.12259/) (2025)
- [Make Your Training Flexible: Towards Deployment-Efficient Video Models](https://mlsanity.com/paper/2503.14237/) (2025)

## Built on by (most cited first, 2 of 2)

- [FlashEdit: Decoupling Speed, Structure, and Semantics for Precise Image Editing](https://mlsanity.com/paper/2509.22244/) (2025)
- [LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization](https://mlsanity.com/paper/2608.08721/) (2026)

## Abstract (arXiv)

Feature caching has recently emerged as a promising method for diffusion model acceleration. It effectively alleviates the inefficiency problem caused by high computational requirements by caching similar features in the inference process of the diffusion model. In this paper, we analyze existing feature caching methods from the perspective of information utilization, and point out that relying solely on historical information will lead to constrained accuracy and speed performance. And we propose a novel paradigm that introduces future information via self-speculation based on the information similarity at the same time step across different iteration times. Based on this paradigm, we present textitSpecDiff, a training-free multi-level feature caching strategy including a cached feature selection algorithm and a multi-level feature classification algorithm. (1) Feature selection algorithm based on self-speculative information. textitSpecDiff determines a dynamic importance score for each token based on self-speculative information and historical information, and performs cached feature selection through the importance score. (2) Multi-level feature classification algorithm based on feature importance scores. textitSpecDiff classifies tokens by leveraging the differences in feature importance scores and introduces a multi-level feature calculation strategy. Extensive experiments show that textitSpecDiff achieves average 2.80 ×, 2.74 × , and 3.17× speedup with negligible quality loss in Stable Diffusion 3, 3.5, and FLUX compared to RFlow on NVIDIA A800-80GB GPU. By merging speculative and historical information, textitSpecDiff overcomes the speedup-accuracy trade-off bottleneck, pushing the Pareto frontier of speedup and accuracy in the efficient diffusion model inference.