# F3-Pruning: A Training-Free and Generalized Pruning Strategy towards Faster and Finer Text-to-Video Synthesis

arXiv: https://arxiv.org/abs/2312.03459 · HTML: https://mlsanity.com/paper/2312.03459/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 6 Dec 2023.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)
- Efficient LLM inference and serving / [Visual token pruning in multimodal models](https://mlsanity.com/inference/visual-token-pruning-in-multimodal-models/)

## Builds on (most cited first, 30 of 31)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformers](https://mlsanity.com/paper/2204.14217/) (2022)
- [Voxel Transformer for 3D Object Detection](https://mlsanity.com/paper/2109.02497/) (2021)
- [Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer](https://mlsanity.com/paper/2204.03638/) (2022)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [Token Merging for Fast Stable Diffusion](https://mlsanity.com/paper/2303.17604/) (2023)
- [Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon](https://mlsanity.com/paper/1705.07565/) (2017)
- [Siam R-CNN: Visual Tracking by Re-Detection](https://mlsanity.com/paper/1911.12836/) (2019)
- [Video Generation From Text](https://mlsanity.com/paper/1710.00421/) (2017)
- [Group Fisher Pruning for Practical Network Compression](https://mlsanity.com/paper/2108.00708/) (2021)
- [A Signal Propagation Perspective for Pruning Neural Networks at Initialization](https://mlsanity.com/paper/1906.06307/) (2019)
- [To Create What You Tell: Generating Videos from Captions](https://mlsanity.com/paper/1804.08264/) (2018)
- [Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers](https://mlsanity.com/paper/2304.10716/) (2023)
- [Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure](https://mlsanity.com/paper/1904.03837/) (2019)
- [Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning](https://mlsanity.com/paper/2002.04809/) (2020)
- [Q-ViT: Fully Differentiable Quantization for Vision Transformer](https://mlsanity.com/paper/2201.07703/) (2022)
- [Two-stream Collaborative Learning with Spatial-Temporal Attention for Video Classification](https://mlsanity.com/paper/1711.03273/) (2017)

## Built on by (most cited first, 10 of 10)

- [One-Way Ticket:Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models](https://mlsanity.com/paper/2505.21960/) (2025)
- [FastLightGen: Fast and Light Video Generation with Fewer Steps and Parameters](https://mlsanity.com/paper/2603.01685/) (2026)
- [Training-Free Semantic Video Composition via Pre-trained Diffusion Model](https://mlsanity.com/paper/2401.09195/) (2024)
- [Attention in Diffusion Model: A Survey](https://mlsanity.com/paper/2504.03738/) (2025)
- [DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers](https://mlsanity.com/paper/2602.16968/) (2026)
- [Any Target Can be Offense: Adversarial Example Generation via Generalized Latent Infection](https://mlsanity.com/paper/2407.12292/) (2024)
- [FastSTAR: Spatiotemporal Token Pruning for Efficient Autoregressive Video Synthesis](https://mlsanity.com/paper/2603.07192/) (2026)
- [V.I.P. : Iterative Online Preference Distillation for Efficient Video Diffusion Models](https://mlsanity.com/paper/2508.03254/) (2025)
- [PARE: Pruning and Adaptive Routing for Efficient Video Generation](https://mlsanity.com/paper/2605.27336/) (2026)
- [VDOT++: Unified Few-Step Video Generation via Unbalanced Optimal Transport Distillation](https://mlsanity.com/paper/2610.03221/) (2026)

## Abstract (arXiv)

Recently Text-to-Video (T2V) synthesis has undergone a breakthrough by training transformers or diffusion models on large-scale datasets. Nevertheless, inferring such large models incurs huge costs.Previous inference acceleration works either require costly retraining or are model-specific.To address this issue, instead of retraining we explore the inference process of two mainstream T2V models using transformers and diffusion models.The exploration reveals the redundancy in temporal attention modules of both models, which are commonly utilized to establish temporal relations among frames.Consequently, we propose a training-free and generalized pruning strategy called F3-Pruning to prune redundant temporal attention weights.Specifically, when aggregate temporal attention values are ranked below a certain ratio, corresponding weights will be pruned.Extensive experiments on three datasets using a classic transformer-based model CogVideo and a typical diffusion-based model Tune-A-Video verify the effectiveness of F3-Pruning in inference acceleration, quality assurance and broad applicability.