# Token Caching for Diffusion Transformer Acceleration

arXiv: https://arxiv.org/abs/2409.18523 · HTML: https://mlsanity.com/paper/2409.18523/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 46 of its 63 references that are in the map.
- First posted on arXiv 27 Sep 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference](https://mlsanity.com/paper/2310.04378/) (2023)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)

## Built on by (most cited first, 30 of 30)

- [Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Models](https://mlsanity.com/paper/2508.12880/) (2025)
- [FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximation](https://mlsanity.com/paper/2505.20353/) (2025)
- [HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration](https://mlsanity.com/paper/2410.01723/) (2024)
- [Attend to Not Attended: Structure-then-Detail Token Merging for Post-training DiT Acceleration](https://mlsanity.com/paper/2505.11707/) (2025)
- [A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generation](https://mlsanity.com/paper/2510.19755/) (2025)
- [ERTACache: Error Rectification and Timesteps Adjustment for Efficient Diffusion](https://mlsanity.com/paper/2508.21091/) (2025)
- [SparseDiT: Token Sparsification for Efficient Diffusion Transformer](https://mlsanity.com/paper/2412.06028/) (2024)
- [Training-free Mixed-Resolution Latent Upsampling for Spatially Accelerated Diffusion Transformers](https://mlsanity.com/paper/2507.08422/) (2025)
- [TC-Pad'e: Trajectory-Consistent Pad'e Approximation for Diffusion Acceleration](https://mlsanity.com/paper/2603.02943/) (2026)
- [SuperGen: An Efficient Ultra-high-resolution Video Generation System with Sketching and Tiling](https://mlsanity.com/paper/2508.17756/) (2025)
- [RAPID^3: Tri-Level Reinforced Acceleration Policies for Diffusion Transformer](https://mlsanity.com/paper/2509.22323/) (2025)
- [Motion-Aware Caching for Efficient Autoregressive Video Generation](https://mlsanity.com/paper/2605.01725/) (2026)
- [One-Way Ticket:Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models](https://mlsanity.com/paper/2505.21960/) (2025)
- [DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers](https://mlsanity.com/paper/2602.16968/) (2026)
- [From Cradle to Cane: A Two-Pass Framework for High-Fidelity Lifespan Face Aging](https://mlsanity.com/paper/2506.20977/) (2025)
- [Post-Training Quantization for Audio Diffusion Transformers](https://mlsanity.com/paper/2510.00313/) (2025)
- [FEB-Cache: Frequency-Guided Exposure Bias Reduction for Enhancing Diffusion Transformer Caching](https://mlsanity.com/paper/2503.07120/) (2025)
- [Adaptive Hybrid Caching for Efficient Text-to-Video Diffusion Model Acceleration](https://mlsanity.com/paper/2508.12691/) (2025)
- [Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer](https://mlsanity.com/paper/2511.19778/) (2025)
- [Fast-ARDiff: An Entropy-informed Acceleration Framework for Continuous Space Autoregressive Generation](https://mlsanity.com/paper/2512.08537/) (2025)
- [No Cache Left Idle: Accelerating diffusion model via Extreme-slimming Caching](https://mlsanity.com/paper/2512.12604/) (2025)
- [Token Pruning for In-Context Generation in Diffusion Transformers](https://mlsanity.com/paper/2602.01609/) (2026)
- [Can We Achieve Efficient Diffusion without Self-Attention? Distilling Self-Attention into Convolutions](https://mlsanity.com/paper/2504.21292/) (2025)
- [DSTAR: Accelerating Diffusion Transformers via Spatial and Temporal Redundancy Reduction](https://mlsanity.com/paper/2607.15846/) (2026)
- [InvarDiff: Cross-Scale Invariance Caching for Accelerated Diffusion Models](https://mlsanity.com/paper/2512.05134/) (2025)
- [Drift-AR: Single-Step Visual Autoregressive Generation via Anti-Symmetric Drifting](https://mlsanity.com/paper/2603.28049/) (2026)
- [Dual-Rate Diffusion: Accelerating diffusion models with an interleaved heavy-light network](https://mlsanity.com/paper/2605.18190/) (2026)
- [Trajectory-Consistent Calibration for Cache-Accelerated Diffusion Models](https://mlsanity.com/paper/2605.24870/) (2026)
- [BRACE: Taming Sharp Irregularities via Barycentric Rational Forecasting for Fast Diffusion Transformers Inference](https://mlsanity.com/paper/2608.07572/) (2026)
- [Memory-Efficient Training-Free Acceleration of Diffusion Transformers with BaryCache](https://mlsanity.com/paper/2608.28670/) (2026)

## Abstract (arXiv)

Diffusion transformers have gained substantial interest in diffusion generative modeling due to their outstanding performance. However, their computational demands, particularly the quadratic complexity of attention mechanisms and multi-step inference processes, present substantial bottlenecks that limit their practical applications. To address these challenges, we propose TokenCache, a novel acceleration method that leverages the token-based multi-block architecture of transformers to reduce redundant computations. TokenCache tackles three critical questions: (1) Which tokens should be pruned and reused by the caching mechanism to eliminate redundancy? (2) Which blocks should be targeted for efficient caching? (3) At which time steps should caching be applied to balance speed and quality? In response to these challenges, TokenCache introduces a Cache Predictor that hierarchically addresses these issues by (1) Token pruning: assigning importance scores to each token to determine which tokens to prune and reuse; (2) Block selection: allocating pruning ratio to each block to adaptively select blocks for caching; (3) Temporal Scheduling: deciding at which time steps to apply caching strategies. Experimental results across various models demonstrate that TokenCache achieves an effective trade-off between generation quality and inference speed for diffusion transformers.