mlsanityData to 14 Oct 2026

LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers

arXiv 2412.12444 ↗First posted 17 Dec 2024cs.LG cs.AIcited by 37

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion Transformers have emerged as the preeminent models for a wide array of generative tasks, demonstrating superior performance and efficacy across various applications. The promising results come at the cost of slow inference, as each denoising step requires running the whole transformer model with a large amount of parameters. In this paper, we show that performing the full computation of the model at each diffusion step is unnecessary, as some computations can be skipped by lazily reusing the results of previous steps. Furthermore, we show that the lower bound of similarity between outputs at consecutive steps is notably high, and this similarity can be linearly approximated using the inputs. To verify our demonstrations, we propose the LazyDiT, a lazy learning framework that efficiently leverages cached results from earlier steps to skip redundant computations. Specifically, we incorporate lazy learning layers into the model, effectively trained to maximize laziness, enabling dynamic skipping of redundant computations. Experimental results show that LazyDiT outperforms the DDIM sampler across multiple diffusion transformer models at various resolutions. Furthermore, we implement our method on mobile devices, achieving better performance than DDIM with similar latency. Code: https://github.com/shawnricecake/lazydit

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 76 of its 140 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

76 of 140 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Pruning then Reweighting: Towards Data-Efficient Training of Diffusion Models27 Sep
  2. 2024Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching3 Jun
  3. 2024Unraveling the Smoothness Properties of Diffusion Models: A Gaussian Mixture Perspective26 May
  4. 2024LD-Pruner: Efficient Pruning of Latent Diffusion Models using Task-Agnostic Insights18 Apr
  5. 2024LAPTOP-Diff: Layer Pruning and Normalized Distillation for Compressing Diffusion Models17 Apr
  6. 2024SparseDM: Toward Sparse Efficient Diffusion Models16 Apr
  7. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillation21 Feb
  8. 2023Cache Me if You Can: Accelerating Diffusion Models through Block Caching6 Dec
  9. 2023One-step Diffusion with Distribution Matching Distillation30 Nov
  10. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference6 Oct
  11. 2023PTQD: Accurate Post-Training Quantization for Diffusion Models18 May
  12. 2022All are Worth Words: A ViT Backbone for Diffusion Models25 Sep
5 more
  1. 2022Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Model16 Aug
  2. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps2 Jun
  3. 2022Pseudo Numerical Methods for Diffusion Models on Manifolds20 Feb
  4. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  5. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 59

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2016Improved Techniques for Training GANscited by 2,970
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  7. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  8. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  9. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  10. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  11. 2021Generating Images with Sparse Representationscited by 266
  12. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 243
38 more
  1. 2016Robust Estimators in High Dimensions without the Computational Intractabilitycited by 209
  2. 2021SPViT: Enabling Faster Vision Transformers via Soft Token Pruningcited by 167
  3. 2021Meta-learning via Language Model In-context Tuningcited by 164
  4. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 103
  5. 2018Solving Linear Programs in the Current Matrix Multiplication Timecited by 100
  6. 2019Solving Empirical Risk Minimization in the Current Matrix Multiplication Timecited by 53
  7. 2020A Faster Interior Point Method for Semidefinite Programmingcited by 53
  8. 2020Faster Dynamic Matrix Inverse for Faster LPscited by 44
  9. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  10. 2023How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computationcited by 44
  11. 2019A Deterministic Linear Program Solver in Current Matrix Multiplication Timecited by 42
  12. 2019Quantum Entropy Scoring for Fast Robust Mean Estimation and Improved Outlier Detectioncited by 41
  13. 2023Randomized and Deterministic Attention Sparsification Algorithms for Over-parameterized Feature Dimensioncited by 37
  14. 2022A Faster Small Treewidth SDP Solvercited by 36
  15. 2018Memory-based Parameter Adaptationcited by 35
  16. 2020Robust Sub-Gaussian Principal Component Analysis and Width-Independent Schatten Packingcited by 35
  17. 2023An Iterative Algorithm for Rescaled Hyperbolic Functions Regressioncited by 35
  18. 2021Achieving on-Mobile Real-Time Super-Resolution with Neural Architecture and Pruning Searchcited by 34
  19. 2021Solving SDP Faster: A Robust IPM Framework and Efficient Implementationcited by 33
  20. 2023Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edgecited by 33
  21. 2022Pruning-as-Search: Efficient Neural Architecture Search via Channel Pruning and Structural Reparameterizationcited by 32
  22. 2014Stochastic Primal-Dual Coordinate Method for Regularized Empirical Risk Minimizationcited by 31
  23. 2020A Nearly-Linear Time Algorithm for Linear Programs with Small Treewidth: A Multiscale Representation of Robust Central Pathcited by 29
  24. 2022Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Trainingcited by 24
  25. 2024OmniControlNet: Dual-stage Integration for Conditional Image Generationcited by 24
  26. 2023Dolfin: Diffusion Layout Transformers without Autoencodercited by 23
  27. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  28. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 22
  29. 2020Solving Tall Dense Linear Programs in Nearly Linear Timecited by 21
  30. 2014Competing with the Empirical Risk Minimizer in a Single Passcited by 20
  31. 2022Compiler-Aware Neural Architecture Search for On-Mobile Real-time Super-Resolutioncited by 20
  32. 2024Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrixcited by 20
  33. 2024Pruning Foundation Models for High Accuracy without Retrainingcited by 20
  34. 2024Looped ReLU MLPs May Be All You Need as Practical Programmable Computerscited by 19
  35. 2024Differential Privacy Mechanisms in Neural Tangent Kernel Regressioncited by 14
  36. 2024Fast and Memory-Efficient Video Diffusion Using Streamlined Inferencecited by 14
  37. 2018On the Local Minima of the Empirical Riskcited by 12
  38. 2023A Unified Scheme of ResNet and Softmaxcited by 12

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 30; 2026: 7.

37 papers.

Sort:

By area:

  1. 2025OminiControl2: Efficient Conditioning for Diffusion Transformerscited by 57
  2. 2025FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximationcited by 24
  3. 2025VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Votingcited by 22
  4. 2025T2VWorldBench: A Benchmark for Evaluating World Knowledge in Text-to-Video Generationcited by 19
  5. 2025Taming Diffusion for Dataset Distillation with High Representativenesscited by 18
  6. 2025A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generationcited by 16
  7. 2025High-Order Matching for One-Step Shortcut Diffusion Modelscited by 15
  8. 2025AB-Cache: Training-Free Acceleration of Diffusion Models via Adams-Bashforth Cached Feature Reusecited by 13
  9. 2025Structured Agent Distillation for Large Language Modelcited by 13
  10. 2025Force Matching with Relativistic Constraints: A Physics-Inspired Approach to Stable and Efficient Generative Modelingcited by 12
  11. 2025Integrating Large Language Models for Automated Structural Analysiscited by 12
  12. 2025ERTACache: Error Rectification and Timesteps Adjustment for Efficient Diffusioncited by 12
25 more
  1. 2025Circuit Complexity Bounds for Visual Autoregressive Modelcited by 10
  2. 2025On Computational Limits of FlowAR Models: Expressivity and Efficiencycited by 10
  3. 2025Theoretical Guarantees for High Order Trajectory Refinement in Generative Flowscited by 10
  4. 2025FastCar: Cache Attentive Replay for Fast Auto-Regressive Video Generation on the Edgecited by 10
  5. 2025Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transformcited by 9
  6. 2025QuartDepth: Post-Training Quantization for Real-Time Depth Estimation on the Edgecited by 8
  7. 2026Shallow-π: Knowledge Distillation for Flow-based VLAscited by 8
  8. 2025Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapsecited by 7
  9. 2025Automating Structural Engineering Workflows with Large Language Model Agentscited by 6
  10. 2025HDCompression: Hybrid-Diffusion Image Compression for Ultra-Low Bitratescited by 5
  11. 2025Open-Source Multimodal Moxin Models with Moxin-VLM and Moxin-VLAcited by 5
  12. 2025Neural Algorithmic Reasoning for Hypergraphs with Looped Transformerscited by 4
  13. 2025UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depthscited by 4
  14. 2025Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approachescited by 4
  15. 2025HOFAR: High-Order Augmentation of Flow Autoregressive Transformerscited by 4
  16. 2025Sortblock: Similarity-Aware Feature Reuse for Diffusion Modelcited by 3
  17. 2025Collaborative Compression for Large-Scale MoE Deployment on Edgecited by 3
  18. 2026AdaCorrection: Adaptive Offset Cache Correction for Accurate Diffusion Transformerscited by 3
  19. 2025Theoretical Foundation of Flow-Based Time Series Generation: Provable Approximation, Generalization, and Efficiencycited by 2
  20. 2025Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMscited by 2
  21. 2026Where to Refine, When to Stop: Rethinking Redundancy via Latent Discrepancy for Efficient Visual Autoregressive Generationcited by 0
  22. 2026Fine-Tuning Qwen3-27B for C-to-Rust Code Translation: A Three-Stage Curriculum of Pretraining, Debugging-Aware SFT, and Task-Specific SFTcited by 0
  23. 2026LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perceptioncited by 0
  24. 2026From Patching to Pruning Visual Computation in Vision Language Modelscited by 0
  25. 2026PixARG: Pixel-Asynchronous Retrieval-Augmented Diffusion for Fast and Realistic Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DiffSparse: Accelerating Diffusion Transformers with Learned Token Sparsity
  2. 2024TinyFusion: Diffusion Transformers Learned Shallow
  3. 2024SparseDiT: Token Sparsification for Efficient Diffusion Transformer
  4. 2025BlockDance: Reuse Structurally Similar Spatio-Temporal Features to Accelerate Diffusion Transformers
  5. 2025Exploring Magnitude Preservation and Rotation Modulation in Diffusion Transformers
  6. 2024FasterDiT: Towards Faster Diffusion Transformers Training without Architecture Modification
  7. 2026Elastic Diffusion Transformer
  8. 2024HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration