# FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximation

arXiv: https://arxiv.org/abs/2505.20353 · HTML: https://mlsanity.com/paper/2505.20353/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 39 of its 45 references that are in the map.
- First posted on arXiv 26 May 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 39)

- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://mlsanity.com/paper/2404.14618/) (2024)
- [DeepCache: Accelerating Diffusion Models for Free](https://mlsanity.com/paper/2312.00858/) (2023)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [Confident Adaptive Language Modeling](https://mlsanity.com/paper/2207.07061/) (2022)
- [Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model](https://mlsanity.com/paper/2411.19108/) (2024)
- [Depth-Adaptive Transformer](https://mlsanity.com/paper/1910.10073/) (2019)
- [SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning](https://mlsanity.com/paper/2012.09852/) (2020)
- [Token Merging for Fast Stable Diffusion](https://mlsanity.com/paper/2303.17604/) (2023)
- [Real-Time Video Generation with Pyramid Attention Broadcast](https://mlsanity.com/paper/2408.12588/) (2024)
- [Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching](https://mlsanity.com/paper/2406.01733/) (2024)
- [IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers](https://mlsanity.com/paper/2106.12620/) (2021)
- [Token Fusion: Bridging the Gap between Token Pruning and Token Merging](https://mlsanity.com/paper/2312.01026/) (2023)
- [FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality](https://mlsanity.com/paper/2410.19355/) (2024)
- [Adaptive Caching for Faster Video Generation with Diffusion Transformers](https://mlsanity.com/paper/2411.02397/) (2024)
- [Compressing Large-Scale Transformer-Based Models: A Case Study on BERT](https://mlsanity.com/paper/2002.11985/) (2020)
- [LooseControl: Lifting ControlNet for Generalized Depth Conditioning](https://mlsanity.com/paper/2312.03079/) (2023)
- [AdaNet: Adaptive Structural Learning of Artificial Neural Networks](https://mlsanity.com/paper/1607.01097/) (2016)
- [Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis](https://mlsanity.com/paper/2304.03869/) (2023)

## Built on by (most cited first, 24 of 24)

- [A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generation](https://mlsanity.com/paper/2510.19755/) (2025)
- [ERTACache: Error Rectification and Timesteps Adjustment for Efficient Diffusion](https://mlsanity.com/paper/2508.21091/) (2025)
- [BWCache: Accelerating Video Diffusion Transformers through Block-Wise Caching](https://mlsanity.com/paper/2509.13789/) (2025)
- [TinyServe: Query-Aware Cache Selection for Efficient LLM Serving](https://mlsanity.com/paper/2509.12211/) (2025)
- [Efficient Video Diffusion Models: Advancements and Challenges](https://mlsanity.com/paper/2604.15911/) (2026)
- [PiKV: KV Cache Management System for Mixture of Experts](https://mlsanity.com/paper/2508.06526/) (2025)
- [Frequency-Aware Error-Bounded Caching for Accelerating Diffusion Transformers](https://mlsanity.com/paper/2603.05315/) (2026)
- [1.x-Distill: Breaking the Diversity, Quality, and Efficiency Barrier in Distribution Matching Distillation](https://mlsanity.com/paper/2604.04018/) (2026)
- [SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling](https://mlsanity.com/paper/2508.15190/) (2025)
- [AdaCorrection: Adaptive Offset Cache Correction for Accurate Diffusion Transformers](https://mlsanity.com/paper/2602.13357/) (2026)
- [Accelerating Diffusion-based Video Editing via Heterogeneous Caching: Beyond Full Computing at Sampled Denoising Timestep](https://mlsanity.com/paper/2603.24260/) (2026)
- [CachedSearch: Training-Free Cached Exploration for Test-Time Search in Video Diffusion](https://mlsanity.com/paper/2607.23159/) (2026)
- [Plug-and-Play Fidelity Optimization for Diffusion Transformer Acceleration via Cumulative Error Minimization](https://mlsanity.com/paper/2512.23258/) (2025)
- [Anomaly Detection and Early Warning Mechanism for Intelligent Monitoring Systems in Multi-Cloud Environments Based on LLM](https://mlsanity.com/paper/2506.07407/) (2025)
- [QuickMerge++: Fast Token Merging with Autoregressive Prior](https://mlsanity.com/paper/2508.13204/) (2025)
- [Accelerating Frequency Domain Diffusion Models with Error-Feedback Event-Driven Caching](https://mlsanity.com/paper/2604.22901/) (2026)
- [SAFE-DiT: Semantics-Aware Fast-path Execution for High-Resolution Diffusion Transformers](https://mlsanity.com/paper/2606.29360/) (2026)
- [OnlineCache: Learning Dynamic Caching Policies with Error Correction for Efficient Diffusion Inference](https://mlsanity.com/paper/2607.29398/) (2026)
- [Cognitive Load Traces as Symbolic and Visual Accounts of Deep Model Cognition](https://mlsanity.com/paper/2510.15980/) (2025)
- [Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scoping](https://mlsanity.com/paper/2602.06850/) (2026)
- [Test-time Sparsity for Extreme Fast Action Diffusion](https://mlsanity.com/paper/2605.13316/) (2026)
- [HSMLA: Hierarchical Softmax Multi-scale Linear Attention for Efficient Vision Transformers](https://mlsanity.com/paper/2608.07616/) (2026)
- [Magnitude-Direction Decoupling for Fast Video Generation with Flow Matching Models](https://mlsanity.com/paper/2608.17695/) (2026)
- [Koopman Observers for Diffusion Acceleration: Correcting Feature Forecasts with Shallow Measurements](https://mlsanity.com/paper/2610.10366/) (2026)

## Abstract (arXiv)

Diffusion Transformers (DiT) are powerful generative models but remain computationally intensive due to their iterative structure and deep transformer stacks. To alleviate this inefficiency, we propose FastCache, a hidden-state-level caching and compression framework that accelerates DiT inference by exploiting redundancy within the model's internal representations. FastCache introduces a dual strategy: (1) a spatial-aware token selection mechanism that adaptively filters redundant tokens based on hidden-state saliency, and (2) a transformer-level cache that reuses latent activations across timesteps when changes fall below a predefined threshold. These modules work jointly to reduce unnecessary computation while preserving generation fidelity through learnable linear approximation. Theoretical analysis shows that FastCache maintains bounded approximation error under a hypothesis-testing-based decision rule. Empirical evaluations across multiple DiT variants demonstrate substantial reductions in latency and memory usage, achieving the best generation quality among existing cache methods, as measured by FID and t-FID. To further improve the speedup of FastCache, we also introduce a token merging module that merges redundant tokens based on k-NN density. Code is available at hrefhttps://github.com/NoakLiu/FastCache-xDiThttps://github.com/NoakLiu/FastCache-xDiT.