# FORA: Fast-Forward Caching in Diffusion Transformer Acceleration

arXiv: https://arxiv.org/abs/2407.01425 · HTML: https://mlsanity.com/paper/2407.01425/

## Key facts

- Cited by 137 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 30 of its 33 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [Scaling up GANs for Text-to-Image Synthesis](https://mlsanity.com/paper/2303.05511/) (2023)
- [DeepCache: Accelerating Diffusion Models for Free](https://mlsanity.com/paper/2312.00858/) (2023)
- [Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.06503/) (2022)
- [PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation](https://mlsanity.com/paper/2403.04692/) (2024)
- [Post-training Quantization on Diffusion Models](https://mlsanity.com/paper/2211.15736/) (2022)
- [Structural Pruning for Diffusion Models](https://mlsanity.com/paper/2305.10924/) (2023)
- [PTQD: Accurate Post-Training Quantization for Diffusion Models](https://mlsanity.com/paper/2305.10657/) (2023)
- [DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics](https://mlsanity.com/paper/2310.13268/) (2023)
- [Cache Me if You Can: Accelerating Diffusion Models through Block Caching](https://mlsanity.com/paper/2312.03209/) (2023)
- [PIXART-δ: Fast and Controllable Image Generation with Latent Consistency Models](https://mlsanity.com/paper/2401.05252/) (2024)
- [Faster Diffusion via Temporal Attention Decomposition](https://mlsanity.com/paper/2404.02747/) (2024)
- [Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models](https://mlsanity.com/paper/2405.05252/) (2024)
- [DREAM: Diffusion Rectification and Estimation-Adaptive Models](https://mlsanity.com/paper/2312.00210/) (2023)
- [Accelerating Text-to-Image Editing via Cache-Enabled Sparse Diffusion Inference](https://mlsanity.com/paper/2305.17423/) (2023)

## Built on by (most cited first, 30 of 137)

- [Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model](https://mlsanity.com/paper/2411.19108/) (2024)
- [dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching](https://mlsanity.com/paper/2506.06295/) (2025)
- [Accelerating Diffusion Transformers with Token-wise Feature Caching](https://mlsanity.com/paper/2410.05317/) (2024)
- [From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeers](https://mlsanity.com/paper/2503.06923/) (2025)
- [Adaptive Caching for Faster Video Generation with Diffusion Transformers](https://mlsanity.com/paper/2411.02397/) (2024)
- [EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models](https://mlsanity.com/paper/2506.10100/) (2025)
- [OminiControl2: Efficient Conditioning for Diffusion Transformers](https://mlsanity.com/paper/2503.08280/) (2025)
- [EEdit: Rethinking the Spatial and Temporal Redundancy for Efficient Image Editing](https://mlsanity.com/paper/2503.10270/) (2025)
- [Less is Enough: Training-Free Video Diffusion Acceleration via Runtime-Adaptive Caching](https://mlsanity.com/paper/2507.02860/) (2025)
- [SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruning](https://mlsanity.com/paper/2509.05614/) (2025)
- [Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animation](https://mlsanity.com/paper/2509.16630/) (2025)
- [FlashVSR: Towards Real-Time Diffusion-Based Streaming Video Super-Resolution](https://mlsanity.com/paper/2510.12747/) (2025)
- [DiCache: Let Diffusion Model Determine Its Own Cache](https://mlsanity.com/paper/2508.17356/) (2025)
- [MagCache: Fast Video Generation with Magnitude-Aware Cache](https://mlsanity.com/paper/2506.09045/) (2025)
- [Efficient Diffusion Models: A Survey](https://mlsanity.com/paper/2502.06805/) (2025)
- [Token Caching for Diffusion Transformer Acceleration](https://mlsanity.com/paper/2409.18523/) (2024)
- [StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generation](https://mlsanity.com/paper/2511.07399/) (2025)
- [SpeCa: Accelerating Diffusion Transformers with Speculative Feature Caching](https://mlsanity.com/paper/2509.11628/) (2025)
- [Rethinking Token-wise Feature Caching: Accelerating Diffusion Transformers with Dual Feature Caching](https://mlsanity.com/paper/2412.18911/) (2024)
- [Flow caching for autoregressive video generation](https://mlsanity.com/paper/2602.10825/) (2026)
- [SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformers](https://mlsanity.com/paper/2411.10510/) (2024)
- [DiTFastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformers](https://mlsanity.com/paper/2503.22796/) (2025)
- [FreqCa: Accelerating Diffusion Models via Frequency-Aware Caching](https://mlsanity.com/paper/2510.08669/) (2025)
- [Compute Only 16 Tokens in One Timestep: Accelerating Diffusion Transformers with Cluster-Driven Feature Caching](https://mlsanity.com/paper/2509.10312/) (2025)
- [FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformers](https://mlsanity.com/paper/2506.04213/) (2025)
- [OmniCache: A Trajectory-Oriented Global Perspective on Training-Free Cache Reuse for Diffusion Transformer Models](https://mlsanity.com/paper/2508.16212/) (2025)
- [HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration](https://mlsanity.com/paper/2410.01723/) (2024)
- [CAT Pruning: Cluster-Aware Token Pruning For Text-to-Image Diffusion Models](https://mlsanity.com/paper/2502.00433/) (2025)
- [A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generation](https://mlsanity.com/paper/2510.19755/) (2025)
- [Accelerating Diffusion Transformer via Gradient-Optimized Cache](https://mlsanity.com/paper/2503.05156/) (2025)

## Abstract (arXiv)

Diffusion transformers (DiT) have become the de facto choice for generating high-quality images and videos, largely due to their scalability, which enables the construction of larger models for enhanced performance. However, the increased size of these models leads to higher inference costs, making them less attractive for real-time applications. We present Fast-FORward CAching (FORA), a simple yet effective approach designed to accelerate DiT by exploiting the repetitive nature of the diffusion process. FORA implements a caching mechanism that stores and reuses intermediate outputs from the attention and MLP layers across denoising steps, thereby reducing computational overhead. This approach does not require model retraining and seamlessly integrates with existing transformer-based diffusion models. Experiments show that FORA can speed up diffusion transformers several times over while only minimally affecting performance metrics such as the IS Score and FID. By enabling faster processing with minimal trade-offs in quality, FORA represents a significant advancement in deploying diffusion transformers for real-time applications. Code will be made publicly available at: https://github.com/prathebaselva/FORA.