# Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference

arXiv: https://arxiv.org/abs/2312.09608 · HTML: https://mlsanity.com/paper/2312.09608/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 49 of its 55 references that are in the map.
- First posted on arXiv 15 Dec 2023.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [Adversarial Diffusion Distillation](https://mlsanity.com/paper/2311.17042/) (2023)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [Label-Efficient Semantic Segmentation with Diffusion Models](https://mlsanity.com/paper/2112.03126/) (2021)
- [Exploiting Diffusion Prior for Real-World Image Super-Resolution](https://mlsanity.com/paper/2305.07015/) (2023)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [DeepCache: Accelerating Diffusion Models for Free](https://mlsanity.com/paper/2312.00858/) (2023)

## Built on by (most cited first, 30 of 34)

- [Hash3D: Training-free Acceleration for 3D Generation](https://mlsanity.com/paper/2404.06091/) (2024)
- [Stage-wise Dynamics of Classifier-Free Guidance in Diffusion Models](https://mlsanity.com/paper/2509.22007/) (2025)
- [A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generation](https://mlsanity.com/paper/2510.19755/) (2025)
- [Distilling Parallel Gradients for Fast ODE Solvers of Diffusion Models](https://mlsanity.com/paper/2507.14797/) (2025)
- [TC-Pad'e: Trajectory-Consistent Pad'e Approximation for Diffusion Acceleration](https://mlsanity.com/paper/2603.02943/) (2026)
- [Foresight: Adaptive Layer Reuse for Accelerated and High-Quality Text-to-Video Generation](https://mlsanity.com/paper/2506.00329/) (2025)
- [Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation](https://mlsanity.com/paper/2506.09376/) (2025)
- [ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation](https://mlsanity.com/paper/2505.21817/) (2025)
- [ResilPhase: Plug-and-Play Phase Mapping and Noise-Resilient Macro-Trajectory Extrapolation for Diffusion Acceleration](https://mlsanity.com/paper/2606.26769/) (2026)
- [Generative Diffusion Receivers: Achieving Pilot-Efficient MIMO-OFDM Communications](https://mlsanity.com/paper/2506.18419/) (2025)
- [TRIM: Scalable 3D Gaussian Diffusion Inference with Temporal and Spatial Trimming](https://mlsanity.com/paper/2511.16642/) (2025)
- [Parallel Diffusion Solver via Residual Dirichlet Policy Optimization](https://mlsanity.com/paper/2512.22796/) (2025)
- [Flow Diverse and Efficient: Learning Momentum Flow Matching via Stochastic Velocity Field Sampling](https://mlsanity.com/paper/2506.08796/) (2025)
- [Predictive Feature Caching for Training-free Acceleration of Molecular Geometry Generation](https://mlsanity.com/paper/2510.04646/) (2025)
- [FasterVAR: Plug-and-Play Acceleration for Visual Autoregressive Models](https://mlsanity.com/paper/2512.16483/) (2025)
- [ProCache: Constraint-Aware Feature Caching with Selective Computation for Diffusion Transformer Acceleration](https://mlsanity.com/paper/2512.17298/) (2025)
- [Not All Layers Are Created Equal: Adaptive LoRA Ranks for Personalized Image Generation](https://mlsanity.com/paper/2603.21884/) (2026)
- [Ctrl-Z Sampling: Scaling Diffusion Sampling with Controlled Random Zigzag Explorations](https://mlsanity.com/paper/2506.20294/) (2025)
- [CPO: Condition Preference Optimization for Controllable Image Generation](https://mlsanity.com/paper/2511.04753/) (2025)
- [TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation](https://mlsanity.com/paper/2603.06057/) (2026)
- [FIS-DiT: Breaking the Few-Step Video Inference Barrier via Training-Free Frame Interleaved Sparsity](https://mlsanity.com/paper/2605.11869/) (2026)
- [OnlineCache: Learning Dynamic Caching Policies with Error Correction for Efficient Diffusion Inference](https://mlsanity.com/paper/2607.29398/) (2026)
- [Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models](https://mlsanity.com/paper/2506.18251/) (2025)
- [ObCLIP: Oblivious CLoud-Device Hybrid Image Generation with Privacy Preservation](https://mlsanity.com/paper/2510.04153/) (2025)
- [PGP-DiffSR: Phase-Guided Progressive Pruning for Efficient Diffusion-based Image Super-Resolution](https://mlsanity.com/paper/2512.02681/) (2025)
- [WaDi: Weight Direction-aware Distillation for One-step Image Synthesis](https://mlsanity.com/paper/2603.08258/) (2026)
- [Backbone-Equated Diffusion OOD via Sparse Internal Snapshots](https://mlsanity.com/paper/2605.11014/) (2026)
- [Three-Step Conditional Diffusion 3D Reconstruction for Light-Field Microscopy](https://mlsanity.com/paper/2605.24959/) (2026)
- [StrideDiffusion: Accelerating Diffusion Models for Time-series Generation](https://mlsanity.com/paper/2607.20545/) (2026)
- [Evaluating Fuzz Testing for Reinforcement Learning Agents](https://mlsanity.com/paper/2607.24577/) (2026)

## Abstract (arXiv)

One of the main drawback of diffusion models is the slow inference time for image generation. Among the most successful approaches to addressing this problem are distillation methods. However, these methods require considerable computational resources. In this paper, we take another approach to diffusion model acceleration. We conduct a comprehensive study of the UNet encoder and empirically analyze the encoder features. This provides insights regarding their changes during the inference process. In particular, we find that encoder features change minimally, whereas the decoder features exhibit substantial variations across different time-steps. This insight motivates us to omit encoder computation at certain adjacent time-steps and reuse encoder features of previous time-steps as input to the decoder in multiple time-steps. Importantly, this allows us to perform decoder computation in parallel, further accelerating the denoising process. Additionally, we introduce a prior noise injection method to improve the texture details in the generated image. Besides the standard text-to-image task, we also validate our approach on other tasks: text-to-video, personalized generation and reference-guided generation. Without utilizing any knowledge distillation technique, our approach accelerates both the Stable Diffusion (SD) and DeepFloyd-IF model sampling by 41% and 24% respectively, and DiT model sampling by 34%, while maintaining high-quality generation performance.