# SlimDiff: Training-Free, Activation-Guided Hands-free Slimming of Diffusion Models

arXiv: https://arxiv.org/abs/2509.21498 · HTML: https://mlsanity.com/paper/2509.21498/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [Q-Diffusion: Quantizing Diffusion Models](https://mlsanity.com/paper/2302.04304/) (2023)
- [SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2403.07378/) (2024)
- [Token Merging for Fast Stable Diffusion](https://mlsanity.com/paper/2303.17604/) (2023)
- [Structural Pruning for Diffusion Models](https://mlsanity.com/paper/2305.10924/) (2023)
- [Revisiting the Nystrom Method for Improved Large-Scale Machine Learning](https://mlsanity.com/paper/1303.1849/) (2013)
- [LAPTOP-Diff: Layer Pruning and Normalized Distillation for Compressing Diffusion Models](https://mlsanity.com/paper/2404.11098/) (2024)
- [MoDeGPT: Modular Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2408.09632/) (2024)
- [Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models](https://mlsanity.com/paper/2405.05252/) (2024)
- [Efficient Diffusion Models: A Survey](https://mlsanity.com/paper/2502.06805/) (2025)
- [SnapGen: Taming High-Resolution Text-to-Image Models for Mobile Devices with Efficient Architectures and Training](https://mlsanity.com/paper/2412.09619/) (2024)
- [Gradient-Free Structured Pruning with Unlabeled Data](https://mlsanity.com/paper/2303.04185/) (2023)
- [Timestep-Aware Correction for Quantized Diffusion Models](https://mlsanity.com/paper/2407.03917/) (2024)
- [Knowledge Distillation of Transformer-based Language Models Revisited](https://mlsanity.com/paper/2206.14366/) (2022)
- [Provably Correct Algorithms for Matrix Column Subset Selection with Selectively Sampled Data](https://mlsanity.com/paper/1505.04343/) (2015)
- [Diffusion Model Quantization: A Review](https://mlsanity.com/paper/2505.05215/) (2025)
- [Swift Cross-Dataset Pruning: Enhancing Fine-Tuning Efficiency in Natural Language Understanding](https://mlsanity.com/paper/2501.02432/) (2025)
- [Improved Fixed-Rank Nystr"om Approximation via QR Decomposition: Practical and Theoretical Aspects](https://mlsanity.com/paper/1708.03218/) (2017)

## Built on by (most cited first, 1 of 1)

- [UnStep: Training-Free Acceleration of Causal Video Diffusion with Fewer Steps Than Distillation](https://mlsanity.com/paper/2609.32518/) (2026)

## Abstract (arXiv)

Diffusion models (DMs), lauded for their generative performance, are computationally prohibitive due to their billion-scale parameters and iterative denoising dynamics. Existing efficiency techniques, such as quantization, timestep reduction, or pruning, offer savings in compute, memory, or runtime but are strictly bottlenecked by reliance on fine-tuning or retraining to recover performance. In this work, we introduce SlimDiff, an automated activation-informed structural compression framework that reduces both attention and feedforward dimensionalities in DMs, while being entirely gradient-free. SlimDiff reframes DM compression as a spectral approximation task, where activation covariances across denoising timesteps define low-rank subspaces that guide dynamic pruning under a fixed compression budget. This activation-aware formulation mitigates error accumulation across timesteps by applying module-wise decompositions over functional weight groups: query--key interactions, value--output couplings, and feedforward projections, rather than isolated matrix factorizations, while adaptively allocating sparsity across modules to respect the non-uniform geometry of diffusion trajectories. SlimDiff achieves up to 35% acceleration and ~100M parameter reduction over baselines, with generation quality on par with uncompressed models without any backpropagation. Crucially, our approach requires only about 500 calibration samples, over 70× fewer than prior methods. To our knowledge, this is the first closed-form, activation-guided structural compression of DMs that is entirely training-free, providing both theoretical clarity and practical efficiency.