# Composition and Alignment of Diffusion Models using Constrained Learning

arXiv: https://arxiv.org/abs/2508.19104 · HTML: https://mlsanity.com/paper/2508.19104/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 51 of its 58 references that are in the map.
- First posted on arXiv 26 Aug 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 51)

- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [Compositional Visual Generation with Composable Diffusion Models](https://mlsanity.com/paper/2206.01714/) (2022)
- [Directly Fine-Tuning Diffusion Models on Differentiable Rewards](https://mlsanity.com/paper/2309.17400/) (2023)
- [Aligning Text-to-Image Models using Human Feedback](https://mlsanity.com/paper/2302.12192/) (2023)
- [Aligning Text-to-Image Diffusion Models with Reward Backpropagation](https://mlsanity.com/paper/2310.03739/) (2023)
- [Human Preference Score: Better Aligning Text-to-Image Models with Human Preference](https://mlsanity.com/paper/2303.14420/) (2023)
- [Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model](https://mlsanity.com/paper/2311.13231/) (2023)
- [Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control](https://mlsanity.com/paper/2409.08861/) (2024)
- [Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control](https://mlsanity.com/paper/2402.15194/) (2024)
- [Video Diffusion Alignment via Reward Gradients](https://mlsanity.com/paper/2407.08737/) (2024)
- [Learning Multi-dimensional Human Preference for Text-to-Image Generation](https://mlsanity.com/paper/2405.14705/) (2024)
- [Aligning Diffusion Models by Optimizing Human Utility](https://mlsanity.com/paper/2404.04465/) (2024)
- [Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review](https://mlsanity.com/paper/2407.13734/) (2024)
- [Constrained Synthesis with Projected Diffusion Models](https://mlsanity.com/paper/2402.03559/) (2024)
- [Classifier-Free Guidance is a Predictor-Corrector](https://mlsanity.com/paper/2408.09000/) (2024)
- [Optimizing DDPM Sampling with Shortcut Fine-Tuning](https://mlsanity.com/paper/2301.13362/) (2023)
- [Diffusion Models Without Attention](https://mlsanity.com/paper/2311.18257/) (2023)
- [Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models](https://mlsanity.com/paper/2405.00760/) (2024)
- [Feedback Efficient Online Fine-Tuning of Diffusion Models](https://mlsanity.com/paper/2402.16359/) (2024)
- [Aligning Optimization Trajectories with Diffusion Models for Constrained Design Generation](https://mlsanity.com/paper/2305.18470/) (2023)

## Built on by (most cited first, 10 of 10)

- [Calibrating Generative Models to Distributional Constraints](https://mlsanity.com/paper/2510.10020/) (2025)
- [Assured autonomy: How operations research powers and orchestrates generative AI systems](https://mlsanity.com/paper/2512.23978/) (2025)
- [Constrained Flow Optimization via Sequential Fine Tuning for Molecular Design](https://mlsanity.com/paper/2605.30610/) (2026)
- [Unlearning in Diffusion Models: A Unified Framework with KL Divergence and Likelihood Constraints](https://mlsanity.com/paper/2605.30825/) (2026)
- [Constrained Diffusion Models with Primal-Dual Inference](https://mlsanity.com/paper/2606.17192/) (2026)
- [Softly Constrained Denoisers for Diffusion Models Applied to Partial Differential Equations](https://mlsanity.com/paper/2512.14980/) (2025)
- [Everywhere Learning: Artificial Intelligence with Pointwise Constraints](https://mlsanity.com/paper/2606.01557/) (2026)
- [Measured Sliders: Learning Continuous Controls from Differentiable Image Measurements](https://mlsanity.com/paper/2609.05234/) (2026)
- [Traversing the Satisfaction-Diversity Frontier in Text-to-Image Diffusion](https://mlsanity.com/paper/2610.02372/) (2026)
- [LLM-to-Flow Distillation: Teaching Natural Language Goals to Scientific Generators](https://mlsanity.com/paper/2610.17246/) (2026)

## Abstract (arXiv)

Diffusion models have become prevalent in generative modeling due to their ability to sample from complex distributions. To improve the quality of generated samples and their compliance with user requirements, two commonly used methods are: (i) Alignment, which involves finetuning a diffusion model to align it with a reward; and (ii) Composition, which combines several pretrained diffusion models together, each emphasizing a desirable attribute in the generated outputs. However, trade-offs often arise when optimizing for multiple rewards or combining multiple models, as they can often represent competing properties. Existing methods cannot guarantee that the resulting model faithfully generates samples with all the desired properties. To address this gap, we propose a constrained optimization framework that unifies alignment and composition of diffusion models by enforcing that the aligned model satisfies reward constraints and/or remains close to each pretrained model. We provide a theoretical characterization of the solutions to the constrained alignment and composition problems and develop a Lagrangian-based primal-dual training algorithm to approximate these solutions. Empirically, we demonstrate our proposed approach in image generation, applying it to alignment and composition, and show that our aligned or composed model satisfies constraints effectively. Our implementation can be found at: hrefhttps://github.com/shervinkhalafi/constrainedcompalignhttps://github.com/shervinkhalafi/constrainedcompalign