mlsanityData to 14 Oct 2026

Training-free Diffusion Model Alignment with Sampling Demons

arXiv 2410.05760 ↗First posted 8 Oct 2024cs.CV cs.AI cs.LGcited by 31

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Aligning diffusion models with user preferences has been a key challenge. Existing methods for aligning diffusion models either require retraining or are limited to differentiable reward functions. To address these limitations, we propose a stochastic optimization approach, dubbed Demon, to guide the denoising process at inference time without backpropagation through reward functions or model retraining. Our approach works by controlling noise distribution in denoising steps to concentrate density on regions corresponding to high rewards through stochastic optimization. We provide comprehensive theoretical and empirical evidence to support and validate our approach, including experiments that use non-differentiable sources of rewards such as Visual-Language Model (VLM) APIs and human judgements. To the best of our knowledge, the proposed approach is the first inference-time, backpropagation-free preference alignment method for diffusion models. Our method can be easily integrated with existing diffusion models without further training. Our experiments show that the proposed approach significantly improves the average aesthetics scores for text-to-image generation. Implementation is available at https://github.com/aiiu-lab/DemonSampling.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 25 of its 40 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

25 of 40 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
  2. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  3. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagation5 Oct
  4. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
  5. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  6. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  11. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  12. 2023Consistency Modelscited by 1,822
7 more
  1. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  2. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  3. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  4. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 846
  5. 2023Universal Guidance for Diffusion Modelscited by 437
  6. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidancecited by 108
  7. 2023Censored Sampling of Diffusion Models Using 3 Minutes of Human Feedbackcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 16.

31 papers.

Sort:

By area:

  1. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
  2. 2025Improving Video Generation with Human Feedbackcited by 284
  3. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 178
  4. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 41
  5. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 40
  6. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  7. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 34
  8. 2025Test-Time Scaling of Diffusion Models via Noise Trajectory Searchcited by 30
  9. 2026Diamond Maps: Efficient Reward Alignment via Stochastic Flow Mapscited by 29
  10. 2025GLASS Flows: Transition Sampling for Alignment of Flow and Diffusion Modelscited by 17
  11. 2026UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generationcited by 13
  12. 2024Free2Guide: Training-Free Text-to-Video Alignment using Image LVLMcited by 9
19 more
  1. 2025Diffusion-Sharpening: Fine-tuning Diffusion Models with Denoising Trajectory Sharpeningcited by 7
  2. 2025PaCo-RL: Advancing Reinforcement Learning for Consistent Image Generation with Pairwise Reward Modelingcited by 7
  3. 2025Inversion-DPO: Precise and Efficient Post-Training for Diffusion Modelscited by 6
  4. 2025TRIM: Scalable 3D Gaussian Diffusion Inference with Temporal and Spatial Trimmingcited by 3
  5. 2026Personalized Image Generation via Human-in-the-loop Bayesian Optimizationcited by 2
  6. 2026Preference Score Distillation: Leveraging 2D Rewards to Align Text-to-3D Generation with Human Preferencecited by 2
  7. 2026Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Modelscited by 1
  8. 2026Latent Reward Registers for Diffusion Preference Alignmentcited by 1
  9. 2025Every Camera Effect, Every Time, All at Once: 4D Gaussian Ray Tracing for Physics-based Camera Effect Data Generationcited by 0
  10. 2026Spectral Evolution Search: Efficient Inference-Time Scaling for Reward-Aligned Image Generationcited by 0
  11. 2026Bridging the Intention-Expression Gap: Aligning Multi-Dimensional Preferences via Hierarchical Relevance Feedback in Text-to-Image Diffusioncited by 0
  12. 2026AMGenC: Generating Charge Balanced Amorphous Materialscited by 0
  13. 2026Towards General Preference Alignment: Diffusion Models at Nash Equilibriumcited by 0
  14. 2026Dependency-Aware Discrete Diffusion for Scene Graph Generationcited by 0
  15. 2026Leveraging Inference-Time Compute for Diffusion Models via Global Scheduling of Denoising Trajectoriescited by 0
  16. 2026Aligning One-Step Generative Models with Reward-Weighted Transport Distillationcited by 0
  17. 2026AutoRef: Harness Optimization for Agentic Multi-Reference Image Generationcited by 0
  18. 2026VIF-Bench: Evaluating Visual Instruction Following in Multi-Reference Image Generationcited by 0
  19. 2026Personalize at Test Time: Learning User Preferences for Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Step-level Denoising-time Diffusion Alignment with Multiple Objectives
  2. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimization
  3. 2025Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference
  4. 2025MIRA: Towards Mitigating Reward Hacking in Inference-Time Alignment of T2I Diffusion Models
  5. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models
  6. 2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards
  7. 2025Inference-Time Alignment of Diffusion Models via Evolutionary Algorithms
  8. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers