# Fast and Memory-Efficient Video Diffusion Using Streamlined Inference

arXiv: https://arxiv.org/abs/2411.01171 · HTML: https://mlsanity.com/paper/2411.01171/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 38 of its 47 references that are in the map.
- First posted on arXiv 2 Nov 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 38)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)
- [Latent Video Diffusion Models for High-Fidelity Long Video Generation](https://mlsanity.com/paper/2211.13221/) (2022)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [DeepCache: Accelerating Diffusion Models for Free](https://mlsanity.com/paper/2312.00858/) (2023)
- [I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models](https://mlsanity.com/paper/2311.04145/) (2023)
- [Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.06503/) (2022)
- [Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2309.15818/) (2023)
- [Q-Diffusion: Quantizing Diffusion Models](https://mlsanity.com/paper/2302.04304/) (2023)
- [Structural Pruning for Diffusion Models](https://mlsanity.com/paper/2305.10924/) (2023)
- [GENIE: Higher-Order Denoising Diffusion Solvers](https://mlsanity.com/paper/2210.05475/) (2022)
- [SimDA: Simple Diffusion Adapter for Efficient Video Generation](https://mlsanity.com/paper/2308.09710/) (2023)
- [Distilling Diffusion Models into Conditional GANs](https://mlsanity.com/paper/2405.05967/) (2024)
- [Advancing Model Pruning via Bi-level Optimization](https://mlsanity.com/paper/2210.04092/) (2022)
- [On the Content Bias in Fr'echet Video Distance](https://mlsanity.com/paper/2404.12391/) (2024)
- [Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models](https://mlsanity.com/paper/2211.02048/) (2022)

## Built on by (most cited first, 14 of 14)

- [LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers](https://mlsanity.com/paper/2412.12444/) (2024)
- [SnapGen-V: Generating a Five-Second Video within Five Seconds on a Mobile Device](https://mlsanity.com/paper/2412.10494/) (2024)
- [Numerical Pruning for Efficient Autoregressive Models](https://mlsanity.com/paper/2412.12441/) (2024)
- [VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Voting](https://mlsanity.com/paper/2507.05116/) (2025)
- [Structured Agent Distillation for Large Language Model](https://mlsanity.com/paper/2505.13820/) (2025)
- [RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation](https://mlsanity.com/paper/2501.04315/) (2025)
- [QuartDepth: Post-Training Quantization for Real-Time Depth Estimation on the Edge](https://mlsanity.com/paper/2503.16709/) (2025)
- [Open-Source Multimodal Moxin Models with Moxin-VLM and Moxin-VLA](https://mlsanity.com/paper/2512.22208/) (2025)
- [Collaborative Compression for Large-Scale MoE Deployment on Edge](https://mlsanity.com/paper/2509.25689/) (2025)
- [LightCache: Memory-Efficient, Training-Free Acceleration for Video Generation](https://mlsanity.com/paper/2510.05367/) (2025)
- [Muninn: Your Trajectory Diffusion Model But Faster](https://mlsanity.com/paper/2605.09999/) (2026)
- [Rethinking Token Reduction for Diffusion Models via Output-Similarity-Awareness](https://mlsanity.com/paper/2605.22011/) (2026)
- [LEAP: Learned Block-wise Evidence Retrieval for Long Audio-Video Perception](https://mlsanity.com/paper/2609.39938/) (2026)
- [From Patching to Pruning Visual Computation in Vision Language Models](https://mlsanity.com/paper/2610.03389/) (2026)

## Abstract (arXiv)

The rapid progress in artificial intelligence-generated content (AIGC), especially with diffusion models, has significantly advanced development of high-quality video generation. However, current video diffusion models exhibit demanding computational requirements and high peak memory usage, especially for generating longer and higher-resolution videos. These limitations greatly hinder the practical application of video diffusion models on standard hardware platforms. To tackle this issue, we present a novel, training-free framework named Streamlined Inference, which leverages the temporal and spatial properties of video diffusion models. Our approach integrates three core components: Feature Slicer, Operator Grouping, and Step Rehash. Specifically, Feature Slicer effectively partitions input features into sub-features and Operator Grouping processes each sub-feature with a group of consecutive operators, resulting in significant memory reduction without sacrificing the quality or speed. Step Rehash further exploits the similarity between adjacent steps in diffusion, and accelerates inference through skipping unnecessary steps. Extensive experiments demonstrate that our approach significantly reduces peak memory and computational overhead, making it feasible to generate high-quality videos on a single consumer GPU (e.g., reducing peak memory of AnimateDiff from 42GB to 11GB, featuring faster inference on 2080Ti).