mlsanityData to 14 Oct 2026

End-to-End Diffusion Latent Optimization Improves Classifier Guidance

arXiv 2303.13703 ↗First posted 23 Mar 2023cs.CV cs.AI cs.LGcited by 108

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Classifier guidance -- using the gradients of an image classifier to steer the generations of a diffusion model -- has the potential to dramatically expand the creative control over image generation and editing. However, currently classifier guidance requires either training new noise-aware models to obtain accurate gradients or using a one-step denoising approximation of the final generation, which leads to misaligned gradients and sub-optimal control. We highlight this approximation's shortcomings and propose a novel guidance method: Direct Optimization of Diffusion Latents (DOODL), which enables plug-and-play guidance by optimizing diffusion latents w.r.t. the gradients of a pre-trained classifier on the true generated pixels, using an invertible diffusion process to achieve memory-efficient backpropagation. Showcasing the potential of more precise guidance, DOODL outperforms one-step classifier guidance on computational and human evaluation metrics across different forms of guidance: using CLIP guidance to improve generations of complex prompts from DrawBench, using fine-grained visual classifiers to expand the vocabulary of Stable Diffusion, enabling image-conditioned generation with a CLIP visual encoder, and improving image aesthetics using an aesthetic scoring network. Code at https://github.com/salesforce/DOODL.

Key facts. Cited by 108 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 39 of its 55 references that are in the map. First posted on arXiv 23 Mar 2023.

Builds on

39 of 55 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Universal Guidance for Diffusion Models14 Feb
  2. 2022Classifier-Free Diffusion Guidance26 Jul
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  4. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  5. 2021Maximum Likelihood Training of Score-Based Diffusion Models22 Jan
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  7. 2020Denoising Diffusion Implicit Models6 Oct
  8. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  12. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
19 more
  1. 2016Density estimation using Real NVPcited by 2,042
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  3. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  4. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  5. 2014NICE: Non-linear Independent Components Estimationcited by 1,058
  6. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  7. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
  8. 2017One pixel attack for fooling deep neural networkscited by 582
  9. 2017Countering Adversarial Images using Input Transformationscited by 520
  10. 2018Invertible Residual Networkscited by 365
  11. 2018Generating Adversarial Examples with Adversarial Networkscited by 258
  12. 2022EDICT: Exact Diffusion Inversion via Coupled Transformationscited by 258
  13. 2018Analyzing Inverse Problems with Invertible Neural Networkscited by 242
  14. 2022Diffusion-based Image Translation using Disentangled Style and Content Representationcited by 149
  15. 2016Memory-Efficient Backpropagation Through Timecited by 78
  16. 2019See Better Before Looking Closer: Weakly Supervised Data Augmentation Network for Fine-Grained Visual Classificationcited by 56
  17. 2021Improving black-box optimization in VAE latent space using decoder uncertaintycited by 51
  18. 2022UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidancecited by 30
  19. 2018Reversible Recurrent Neural Networkscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 18; 2024: 36; 2025: 34; 2026: 20.

108 papers.

Sort:

By area:

  1. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  2. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 448
  3. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 288
  4. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 233
  5. 2023Manifold Preserving Guided Diffusioncited by 157
  6. 2024D-Flow: Differentiating through Flows for Controlled Generationcited by 122
  7. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 119
  8. 2023Optimizing Diffusion Noise Can Serve As Universal Motion Priorscited by 98
  9. 2023Finetuning Text-to-Image Diffusion Models for Fairnesscited by 92
  10. 2024DITTO: Diffusion Inference-Time T-Optimization for Music Generationcited by 73
  11. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 73
  12. 2024Not All Noises Are Created Equally:Diffusion Noise Selection and Optimizationcited by 71
38 more
  1. 2023DiffusionLight: Light Probes for Free by Painting a Chrome Ballcited by 64
  2. 2024Gradient Guidance for Diffusion Models: An Optimization Perspectivecited by 61
  3. 2023On the Design Fundamentals of Diffusion Models: A Surveycited by 60
  4. 2024Deep Reward Supervisions for Tuning Text-to-Image Diffusion Modelscited by 59
  5. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  6. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 55
  7. 2024Diffusion Models and Representation Learning: A Surveycited by 54
  8. 2023Readout Guidance: Learning Control from Diffusion Featurescited by 52
  9. 2024LCM-Lookahead for Encoder-based Text-to-Image Personalizationcited by 48
  10. 2023LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Imagescited by 46
  11. 2024PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Modelscited by 42
  12. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 41
  13. 2024Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimatorscited by 39
  14. 2025Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcingcited by 38
  15. 2024Self-Improving Diffusion Models with Synthetic Datacited by 35
  16. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  17. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  18. 2024Training-free Diffusion Model Alignment with Sampling Demonscited by 31
  19. 2024Improving GFlowNets for Text-to-Image Diffusion Alignmentcited by 27
  20. 2024DyMO: Training-Free Diffusion Model Alignment with Dynamic Multi-Objective Schedulingcited by 27
  21. 2024DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generationcited by 26
  22. 2024Implicit Diffusion: Efficient Optimization through Stochastic Samplingcited by 24
  23. 2025Training-Free Guidance Beyond Differentiability: Scalable Path Steering with Tree Search in Diffusion and Flow Modelscited by 24
  24. 2025Aligning Text to Image in Diffusion Models is Easier Than You Thinkcited by 22
  25. 2023AdaDiff: Adaptive Step Selection for Fast Diffusion Modelscited by 20
  26. 2024SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generationcited by 19
  27. 2023Elucidating The Design Space of Classifier-Guided Diffusion Generationcited by 18
  28. 2023Latent Diffusion Counterfactual Explanationscited by 17
  29. 2023Enhancing Diffusion Models with 3D Perspective Geometry Constraintscited by 16
  30. 2024Towards Reliable Advertising Image Generation Using Human Feedbackcited by 16
  31. 2024Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through f-divergence Minimizationcited by 15
  32. 2024ProCreate, Don't Reproduce! Propulsive Energy Diffusion for Creative Generationcited by 13
  33. 2024Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Modelscited by 13
  34. 2024Diffusion Models as Network Optimizers: Explorations and Analysiscited by 13
  35. 2025Source-Guided Flow Matchingcited by 13
  36. 2024Invertible Diffusion Models for Compressed Sensingcited by 12
  37. 2025CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objectscited by 12
  38. 2025Diverse Text-to-Image Generation via Contrastive Noise Optimizationcited by 12

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Advanced Image Generation: Negative Prompt Optimization and Latent Classifier Guidance
  2. 2024Plug-and-Play Diffusion Distillation
  3. 2025Enhancing Diffusion Model Guidance through Calibration and Regularization
  4. 2026Enhanced Low-Density Region Exploration in Classifier-Guided Diffusion Models Through Modified Reverse Diffusion Sampling
  5. 2025Dynamic Classifier-Free Diffusion Guidance via Online Feedback
  6. 2023Bridging the Gap: Addressing Discrepancies in Diffusion Model Training for Classifier-Free Guidance
  7. 2026Revisiting Classifier-Free Guidance Methods in Latent Diffusion Models
  8. 2022On Distillation of Guided Diffusion Models