mlsanityData to 14 Oct 2026

Understanding LoRA Rank Trade-offs in Diffusion Model Fine-Tuning

arXiv 2609.10656 ↗First posted 9 Sep 2026cs.AI cs.LGcited by 0

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Selecting LoRA rank for diffusion fine-tuning requires balancing quality and compute cost. We present a controlled study on CIFAR-10 using a DDPM U-Net with ranks {2,4,8,16,32}, fixed optimization settings, and a reproducible local-folder pytorch-fid protocol. We report FID, trainable parameters, runtime, and GPU memory, then validate trends with extended-budget DDPM runs (20 epochs; ranks 4/8/16) and a Tiny DiT backbone (10 epochs; ranks 4/8/16). Results show moderate ranks are most efficient: rank 4 achieves the best DDPM FID (124.1380), rank 8 is close (124.2136), and higher ranks provide limited gains despite larger adaptation cost. These findings support small-to-moderate ranks as practical defaults under fixed training budgets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 6 of its 14 references that are in the map. First posted on arXiv 9 Sep 2026.

Builds on

6 of 14 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Scalable Diffusion Models with Transformers19 Dec
  2. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 4

  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  3. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  4. 2025T-LoRA: Single Image Diffusion Model Customization Without Overfittingcited by 15

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CDM-QTA: Quantized Training Acceleration for Efficient LoRA Fine-Tuning of Diffusion Model
  2. 2023LoRA-Enhanced Distillation on Guided Diffusion Models
  3. 2026Beyond Low-Rank Parameterization: Narrowing the Gap Between LoRA and Full Fine-Tuning via Gradient Decomposition
  4. 2025TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision
  5. 2024LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
  6. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?
  7. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuning
  8. 2026When Top-1 Fails: Calibrating LoRA Monitors for Masked Diffusion LMs