mlsanityData to 7 Oct 2026

Source-Prior-Driven Selective Adaptation for Efficient Diffusion Model Finetuning

arXiv 2607.20913 ↗First posted 23 Jul 2026cs.AIcited by 0

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Fine-tuning large diffusion models for new domains or styles involves a trade-off: improving target-specific generation often degrades the pretrained model's broad generative capability. Existing full and parameter-efficient fine-tuning methods typically handle this trade-off only implicitly. In this work, we propose a novel source-prior-driven selective adaptation method to efficiently fine-tune diffusion models, achieving a favorable trade-off. Our method relies on two key observations: (1) the loss of general generative capability is highly inconsistent across pretrained parameters, and (2) parameters that have a relatively small impact on the model's general generative capability remain structurally inconsistent across layers and parameter types. Motivated by these observations, we first learn a static mask to explicitly identify parameters better suited for downstream adaptation, and then construct structured update strategies for the selected subset. Experiments show that our method achieves a better adaptation-retention trade-off than existing strong baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 40 of its 49 references that are in the map. First posted on arXiv 23 Jul 2026.

Builds on

40 of 49 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation10 Sep
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesis5 Mar
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  4. 2020Denoising Diffusion Implicit Models6 Oct
  5. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  10. 2016Learning without Forgettingcited by 2,465
  11. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  12. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
23 more
  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  2. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  3. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  4. 2018Demystifying MMD GANscited by 1,268
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  6. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  7. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  8. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  9. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  10. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  11. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  12. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  13. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  14. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  15. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  16. 2016A survey of sparse representation: algorithms and applicationscited by 58
  17. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
  18. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  19. 2025SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsitycited by 9
  20. 2024KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Modelscited by 8
  21. 2025Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Visioncited by 7
  22. 2025TS-PEFT: Unveiling Token-Level Redundancy in Parameter-Efficient Fine-Tuningcited by 1
  23. 2026ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning
  2. 2025Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model
  3. 2024Diffusion Tuning: Transferring Diffusion Models via Chain of Forgetting
  4. 2025Fine-Tuning Diffusion Models via Intermediate Distribution Shaping
  5. 2025FeRA: Frequency-Energy Constrained Routing for Effective Diffusion Adaptation Fine-Tuning
  6. 2025From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs
  7. 2026Training-Free Adaptation of Diffusion Models via Doob's h-Transform
  8. 2026MMD Guidance: Training-Free Distribution Adaptation for Diffusion Models via Maximum Mean Discrepancy Guidance