Data to 7 Oct 2026

Test-time Alignment of Diffusion Models without Reward Over-optimization

arXiv 2501.05803 ↗First posted 10 Jan 2025cs.LG cs.AI cs.CVcited by 89

On the map: Diffusion models → Diffusion foundations and sampling

also in 1 more

LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Diffusion models excel in generative tasks, but aligning them with specific objectives while maintaining their versatility remains challenging. Existing fine-tuning methods often suffer from reward over-optimization, while approximate guidance approaches fail to optimize target rewards effectively. Addressing these limitations, we propose a training-free, test-time method based on Sequential Monte Carlo (SMC) to sample from the reward-aligned target distribution. Our approach, tailored for diffusion sampling and incorporating tempering techniques, achieves comparable or superior target rewards to fine-tuning methods while preserving diversity and cross-reward generalization. We demonstrate its effectiveness in single-reward optimization, multi-objective scenarios, and online black-box optimization. This work offers a robust solution for aligning diffusion models with diverse downstream objectives without compromising their general capabilities. Code is available at https://github.com/krafton-ai/DAS.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 57 of its 73 references that are in the map. First posted on arXiv 10 Jan 2025.

Builds on

57 of 73 references.

Closest ancestors · 17

Same direction, newest first

5 more

2021Diffusion Models Beat GANs on Image Synthesis11 May
2021Maximum Likelihood Training of Score-Based Diffusion Models22 Jan
2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
2020Denoising Diffusion Implicit Models6 Oct
2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 40

2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2022Training language models to follow instructions with human feedbackcited by 15,712
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
2020Learning to summarize from human feedbackcited by 2,562
2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
2022Video Diffusion Modelscited by 2,280
2019Fine-Tuning Language Models from Human Preferencescited by 2,124
2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928

28 more

2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,375
2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
2023Training Diffusion Models with Reinforcement Learningcited by 939
2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
2023Aligning Text-to-Image Models using Human Feedbackcited by 418
2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 284
2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
2022Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problemcited by 229
2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
2023Practical and Asymptotically Exact Conditional Sampling in Diffusion Modelscited by 218
2021Path Integral Sampler: a stochastic control approach for samplingcited by 175
2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 136
2024Theoretical guarantees on the best-of-n alignment policycited by 116
2024ARGS: Alignment as Reward-Guided Searchcited by 99
2019Elements of Sequential Monte Carlocited by 60
2024Feedback Efficient Online Fine-Tuning of Diffusion Modelscited by 52
2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 40
2024Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidancecited by 30
2024Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biasescited by 29
2024Measuring Diversity in Co-creative Image Generationcited by 15
2023Censored Sampling of Diffusion Models Using 3 Minutes of Human Feedbackcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 38; 2026: 51.

89 papers.

Sort:

By area:

2025Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Expertscited by 85
2025Reasoning with Sampling: Your Base Model is Smarter Than You Thinkcited by 84
2025Inference-time Scaling of Diffusion Models through Classical Searchcited by 48
2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 40
2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 38
2025Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcingcited by 37
2025Dynamic Search for Inference-Time Alignment in Diffusion Modelscited by 36
2025Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimizationcited by 35
2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 31
2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
2025Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Modelscited by 15
2025Inference-Time Scaling of Discrete Diffusion Models via Importance Weighting and Optimal Proposal Designcited by 12

38 more

2025Dual-Process Image Generationcited by 12
2025Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewardscited by 11
2025Dynamic Classifier-Free Diffusion Guidance via Online Feedbackcited by 11
2025ORIGEN: Zero-Shot 3D Orientation Grounding in Text-to-Image Generationcited by 10
2025Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Designcited by 10
2025ImagerySearch: Adaptive Test-Time Search for Video Generation Beyond Semantic Dependency Constraintscited by 9
2025MIRA: Towards Mitigating Reward Hacking in Inference-Time Alignment of T2I Diffusion Modelscited by 7
2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Modelscited by 6
2025Projected Coupled Diffusion for Test-Time Constrained Joint Generationcited by 6
2025Scale Where It Matters: Training-Free Localized Scaling for Diffusion Modelscited by 6
2026Training-Free Adaptation of Diffusion Models via Doob's h-Transformcited by 6
2025Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Modelscited by 5
2025Reinforced sequential Monte Carlo for amortised samplingcited by 5
2026Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?cited by 5
2025TTOM: Test-Time Optimization and Memorization for Compositional Video Generationcited by 4
2026HyperAlign: Hypernetwork for Efficient Test-Time Alignment of Diffusion Modelscited by 4
2026AEGPO: Adaptive Entropy-Guided Policy Optimization for Diffusion Modelscited by 4
2026Reward-Guided Discrete Diffusion via Clean-Sample Markov Chain for Molecule and Biological Sequence Designcited by 3
2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 3
2026Contrastive Distribution Matching for Amortized Sequential Monte Carlo in Discrete Diffusioncited by 3
2026Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inferencecited by 3
2026Inference-Time Scaling for Joint Audio-Video Generationcited by 3
2025CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Explorationcited by 2
2025Tree Reward-Aligned Search for TReASURe in Masked Diffusion Language Modelscited by 2
2025Diffusion Alignment as Variational Expectation-Maximizationcited by 2
2025Hierarchical Schedule Optimization for Fast and Robust Diffusion Model Samplingcited by 2
2026Personalized Image Generation via Human-in-the-loop Bayesian Optimizationcited by 2
2026Entropy Aware Reward Guidance for Diffusion Language Model Alignmentcited by 2
2026Provably Safe Generative Sampling with Constricting Barrier Functionscited by 2
2026Preference Score Distillation: Leveraging 2D Rewards to Align Text-to-3D Generation with Human Preferencecited by 2
2026Inference-Time Scaling in Diffusion Models through Iterative Partial Refinementcited by 2
2026Stitched Value Model for Diffusion Alignmentcited by 2
2025Ctrl-Z Sampling: Scaling Diffusion Sampling with Controlled Random Zigzag Explorationscited by 1
2025Inference-Time Search Using Side Information for Diffusion-Based Image Reconstructioncited by 1
2025Rethinking Test Time Scaling for Flow-Matching Generative Modelscited by 1
2025ITS3D: Inference-Time Scaling for Text-Guided 3D Diffusion Modelscited by 1
2026DeRaDiff: Denoising Time Realignment of Diffusion Modelscited by 1
2026Tuning Real-World Image Restoration at Inference: A Test-Time Scaling Paradigm for Flow Matching Modelscited by 1

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way