mlsanityData to 7 Oct 2026

DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning

arXiv 2304.06648 ↗First posted 13 Apr 2023cs.CVcited by 82

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion models have proven to be highly effective in generating high-quality images. However, adapting large pre-trained diffusion models to new domains remains an open challenge, which is critical for real-world applications. This paper proposes DiffFit, a parameter-efficient strategy to fine-tune large pre-trained diffusion models that enable fast adaptation to new domains. DiffFit is embarrassingly simple that only fine-tunes the bias term and newly-added scaling factors in specific layers, yet resulting in significant training speed-up and reduced model storage costs. Compared with full fine-tuning, DiffFit achieves 2× training speed-up and only needs to store approximately 0.12% of the total model parameters. Intuitive theoretical analysis has been provided to justify the efficacy of scaling factors on fast adaptation. On 8 downstream datasets, DiffFit achieves superior or competitive performances compared to the full fine-tuning while being more efficient. Remarkably, we show that DiffFit can adapt a pre-trained low-resolution generative model to a high-resolution one by adding minimal cost. Among diffusion-based methods, DiffFit sets a new state-of-the-art FID of 3.02 on ImageNet 512×512 benchmark by fine-tuning only 25 epochs from a public pre-trained ImageNet 256×256 checkpoint while being 30× more training efficient than the closest competitor.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 77 of its 89 references that are in the map. First posted on arXiv 13 Apr 2023.

Builds on

77 of 89 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023Consistency Models2 Mar
  2. 2023Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data14 Feb
  3. 2022Scalable Diffusion Models with Transformers19 Dec
  4. 2022Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions22 Sep
  5. 2022Convergence of denoising diffusion models under the manifold hypothesis10 Aug
  6. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning8 Aug
  7. 2022Convergence for score-based generative modeling with polynomial complexity13 Jun
  8. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  9. 2021Variational Diffusion Models1 Jul
  10. 2021Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modeling1 Jun
  11. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  12. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
2 more
  1. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  2. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  11. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  13. 2022Video Diffusion Modelscited by 2,280
  14. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  15. 2015Variational Inference with Normalizing Flowscited by 2,161
  16. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  17. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  18. 2022Visual Prompt Tuningcited by 1,674
  19. 2020Point Transformercited by 1,462
  20. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  21. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  22. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  23. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  24. 2021Video Swin Transformercited by 1,090
  25. 2019An Introduction to Variational Autoencoderscited by 997
  26. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  27. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  28. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  29. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  30. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  31. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  32. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  33. 2021Transformer in Transformercited by 632
  34. 2017Generative Adversarial Networks: An Overviewcited by 588
  35. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  36. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  37. 2020TransTrack: Multiple Object Tracking with Transformercited by 370
  38. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 36; 2025: 23; 2026: 9.

82 papers.

Sort:

By area:

  1. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  2. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 326
  3. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  4. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  5. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  6. 2023ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Modelscited by 127
  7. 2023DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generationcited by 110
  8. 2023Efficient Dataset Distillation via Minimax Diffusioncited by 81
  9. 2024Dynamic Diffusion Transformercited by 60
  10. 2023Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunitycited by 54
  11. 2024Make a Cheap Scaling: A Self-Cascade Diffusion Model for Higher-Resolution Adaptationcited by 50
  12. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
38 more
  1. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  2. 2023SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editingcited by 37
  3. 2024DiffuseHigh: Training-free Progressive High-Resolution Image Synthesis through Structure Guidancecited by 33
  4. 2023Latent Space Editing in Transformer-Based Flow Matchingcited by 32
  5. 2024AccDiffusion: An Accurate Method for Higher-Resolution Image Generationcited by 32
  6. 2023HiDiffusion: Unlocking Higher-Resolution Creativity and Efficiency in Pretrained Diffusion Modelscited by 27
  7. 2024Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectivescited by 27
  8. 2024Dimba: Transformer-Mamba Diffusion Modelscited by 26
  9. 2024Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimizationcited by 25
  10. 2024ResAdapter: Domain Consistent Resolution Adapter for Diffusion Modelscited by 23
  11. 2024Edify Image: High-Quality Image Generation with Pixel Space Laplacian Diffusion Modelscited by 22
  12. 2024Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmarkcited by 21
  13. 2024Scaling Diffusion Mamba with Bidirectional SSMs for Efficient Image and Video Generationcited by 21
  14. 2024Distribution-Aware Data Expansion with Diffusion Modelscited by 20
  15. 2024Latent Watermark: Inject and Detect Watermarks in Latent Diffusion Spacecited by 19
  16. 2025DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformerscited by 17
  17. 2024Adding Conditional Control to Diffusion Models with Reinforcement Learningcited by 16
  18. 2024Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognitioncited by 16
  19. 2024Is One GPU Enough? Pushing Image Generation at Higher-Resolutions with Foundation Modelscited by 14
  20. 2024CutDiffusion: A Simple, Fast, Cheap, and Strong Diffusion Extrapolation Methodcited by 13
  21. 2024Diffusion Tuning: Transferring Diffusion Models via Chain of Forgettingcited by 12
  22. 2024Structure-Guided Adversarial Training of Diffusion Modelscited by 11
  23. 2024Large Convolutional Model Tuning via Filter Subspacecited by 11
  24. 2024Diffusion Model Patching via Mixture-of-Promptscited by 11
  25. 2024DriveDiTFit: Fine-tuning Diffusion Transformers for Autonomous Drivingcited by 11
  26. 2024Transfer Learning for Diffusion Modelscited by 10
  27. 2025Diversity-Driven Generative Dataset Distillation Based on Diffusion Model with Self-Adaptive Memorycited by 10
  28. 2023Fast Training of Diffusion Transformer with Extreme Masking for 3D Point Clouds Generationcited by 9
  29. 2024BeyondScene: Higher-Resolution Human-Centric Scene Generation With Pretrained Diffusioncited by 9
  30. 2025TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Visioncited by 9
  31. 2025Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Modelcited by 8
  32. 2025Latent Wavelet Diffusion For Ultra-High-Resolution Image Synthesiscited by 6
  33. 2024FineDiffusion: Scaling up Diffusion Models for Fine-grained Image Generation with 10,000 Classescited by 5
  34. 2025Provable Sample-Efficient Transfer Learning Conditional Diffusion Models via Representation Learningcited by 5
  35. 2025Dataset Condensation with Color Compensationcited by 5
  36. 2025ScaleDiff: Higher-Resolution Image Synthesis via Efficient and Model-Agnostic Diffusioncited by 5
  37. 2024Efficient Training of Large Vision Models via Advanced Automated Progressive Learningcited by 4
  38. 2026H3-World: Turning Language Understanding into World Controlcited by 4

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Source-Prior-Driven Selective Adaptation for Efficient Diffusion Model Finetuning
  2. 2022On Distillation of Guided Diffusion Models
  3. 2023Diffusion Models Without Attention
  4. 2024Plug-and-Play Diffusion Distillation
  5. 2025Glance: Accelerating Diffusion Models with 1 Sample
  6. 2023Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models
  7. 2025DiffPro: Joint Timestep and Layer-Wise Precision Optimization for Efficient Diffusion Inference
  8. 2025DogFit: Domain-guided Fine-tuning for Efficient Transfer Learning of Diffusion Models