# Relational Diffusion Distillation for Efficient Image Generation

arXiv: https://arxiv.org/abs/2410.07679 · HTML: https://mlsanity.com/paper/2410.07679/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 1 of its 3 references that are in the map.
- First posted on arXiv 10 Oct 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 1 of 1)

- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)

## Built on by (most cited first, 24 of 24)

- [MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Models](https://mlsanity.com/paper/2412.11549/) (2024)
- [Quantized Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2509.21302/) (2025)
- [Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition](https://mlsanity.com/paper/2502.18510/) (2025)
- [Beyond Optimal Transport: Model-Aligned Coupling for Flow Matching](https://mlsanity.com/paper/2505.23346/) (2025)
- [Accelerating Diffusion Transformer via Error-Optimized Cache](https://mlsanity.com/paper/2501.19243/) (2025)
- [MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation](https://mlsanity.com/paper/2507.04290/) (2025)
- [Training-free Mixed-Resolution Latent Upsampling for Spatially Accelerated Diffusion Transformers](https://mlsanity.com/paper/2507.08422/) (2025)
- [SRKD: Towards Efficient 3D Point Cloud Segmentation via Structure- and Relation-aware Knowledge Distillation](https://mlsanity.com/paper/2506.17290/) (2025)
- [Learning Straight Flows: Variational Flow Matching for Efficient Generation](https://mlsanity.com/paper/2511.17583/) (2025)
- [Denoising as Path Planning: Training-Free Acceleration of Diffusion Models with DPCache](https://mlsanity.com/paper/2602.22654/) (2026)
- [Enhancing Image Generation Fidelity via Progressive Prompts](https://mlsanity.com/paper/2501.07070/) (2025)
- [Multi-party Collaborative Attention Control for Image Customization](https://mlsanity.com/paper/2505.01428/) (2025)
- [MultiAnimate: Pose-Guided Image Animation Made Extensible](https://mlsanity.com/paper/2602.21581/) (2026)
- [Masked Generative Nested Transformers with Decode Time Scaling](https://mlsanity.com/paper/2502.00382/) (2025)
- [DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer](https://mlsanity.com/paper/2505.15133/) (2025)
- [DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers](https://mlsanity.com/paper/2602.16968/) (2026)
- [MMPhysVideo: Physically Plausible Video Generation Through Joint RGB-Perception Modeling](https://mlsanity.com/paper/2604.02817/) (2026)
- [Image Recognition with Online Lightweight Vision Transformer: A Survey](https://mlsanity.com/paper/2505.03113/) (2025)
- [Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer](https://mlsanity.com/paper/2511.19778/) (2025)
- [VeCoR -- Velocity Contrastive Regularization for Flow Matching](https://mlsanity.com/paper/2511.18942/) (2025)
- [ReDiF: Reinforced Distillation for Few Step Diffusion](https://mlsanity.com/paper/2512.22802/) (2025)
- [Annealed Langevin Posterior Sampling (ALPS): A Rapid Algorithm for Image Restoration with Multiscale Energy Models](https://mlsanity.com/paper/2601.02594/) (2026)
- [Teacher-Guided Student Self-Knowledge Distillation Using Diffusion Model](https://mlsanity.com/paper/2602.02107/) (2026)
- [Where to Refine, When to Stop: Rethinking Redundancy via Latent Discrepancy for Efficient Visual Autoregressive Generation](https://mlsanity.com/paper/2606.00310/) (2026)

## Abstract (arXiv)

Although the diffusion model has achieved remarkable performance in the field of image generation, its high inference delay hinders its wide application in edge devices with scarce computing resources. Therefore, many training-free sampling methods have been proposed to reduce the number of sampling steps required for diffusion models. However, they perform poorly under a very small number of sampling steps. Thanks to the emergence of knowledge distillation technology, the existing training scheme methods have achieved excellent results at very low step numbers. However, the current methods mainly focus on designing novel diffusion model sampling methods with knowledge distillation. How to transfer better diffusion knowledge from teacher models is a more valuable problem but rarely studied. Therefore, we propose Relational Diffusion Distillation (RDD), a novel distillation method tailored specifically for distilling diffusion models. Unlike existing methods that simply align teacher and student models at pixel level or feature distributions, our method introduces cross-sample relationship interaction during the distillation process and alleviates the memory constraints induced by multiple sample interactions. Our RDD significantly enhances the effectiveness of the progressive distillation framework within the diffusion model. Extensive experiments on several datasets (e.g., CIFAR-10 and ImageNet) demonstrate that our proposed RDD leads to 1.47 FID decrease under 1 sampling step compared to state-of-the-art diffusion distillation methods and achieving 256x speed-up compared to DDIM strategy. Code is available at https://github.com/cantbebetter2/RDD.