# Enhancing Diffusion Models with Text-Encoder Reinforcement Learning

arXiv: https://arxiv.org/abs/2311.15657 · HTML: https://mlsanity.com/paper/2311.15657/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 48 of its 55 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 48)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)

## Built on by (most cited first, 30 of 30)

- [Improving Video Generation with Human Feedback](https://mlsanity.com/paper/2501.13918/) (2025)
- [OmniGen2: Towards Instruction-Aligned Multimodal Generation](https://mlsanity.com/paper/2506.18871/) (2025)
- [ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization](https://mlsanity.com/paper/2406.04312/) (2024)
- [Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization](https://mlsanity.com/paper/2406.04314/) (2024)
- [LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment](https://mlsanity.com/paper/2412.04814/) (2024)
- [Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models](https://mlsanity.com/paper/2508.09968/) (2025)
- [DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models](https://mlsanity.com/paper/2506.03517/) (2025)
- [Scalable Ranked Preference Optimization for Text-to-Image Generation](https://mlsanity.com/paper/2410.18013/) (2024)
- [Improving GFlowNets for Text-to-Image Diffusion Alignment](https://mlsanity.com/paper/2406.00633/) (2024)
- [Boost Your Human Image Generation Model via Direct Preference Optimization](https://mlsanity.com/paper/2405.20216/) (2024)
- [Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation](https://mlsanity.com/paper/2412.01027/) (2024)
- [Aligning Few-Step Diffusion Models with Dense Reward Difference Learning](https://mlsanity.com/paper/2411.11727/) (2024)
- [Reinforcement Learning for Large Model: A Survey](https://mlsanity.com/paper/2508.08189/) (2025)
- [ShortFT: Diffusion Model Alignment via Shortcut-based Fine-Tuning](https://mlsanity.com/paper/2507.22604/) (2025)
- [Learning User Preferences for Image Generation Model](https://mlsanity.com/paper/2508.08220/) (2025)
- [Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generation](https://mlsanity.com/paper/2601.12401/) (2026)
- [Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment](https://mlsanity.com/paper/2506.01511/) (2025)
- [VERTIGO: Visual Preference Optimization for Cinematic Camera Trajectory Generation](https://mlsanity.com/paper/2604.02467/) (2026)
- [DICE: Distilling Classifier-Free Guidance into Text Embeddings](https://mlsanity.com/paper/2502.03726/) (2025)
- [DreamBoothDPO: Improving Personalized Generation using Direct Preference Optimization](https://mlsanity.com/paper/2505.20975/) (2025)
- [Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generation](https://mlsanity.com/paper/2511.01450/) (2025)
- [GRAN-TED: Generating Robust, Aligned, and Nuanced Text Embedding for Diffusion Models](https://mlsanity.com/paper/2512.15560/) (2025)
- [Low-Biased General Annotated Dataset Generation](https://mlsanity.com/paper/2412.10831/) (2024)
- [D2D: Detector-to-Differentiable Critic for Improved Numeracy in Text-to-Image Generation](https://mlsanity.com/paper/2510.19278/) (2025)
- [TextBoost: Boosting Text Encoder for Personalized Text-to-Image Generation](https://mlsanity.com/paper/2409.08248/) (2024)
- [IPGO: Indirect Prompt Gradient Optimization for Parameter-Efficient Prompt-level Fine-Tuning on Text-to-Image Models](https://mlsanity.com/paper/2503.21812/) (2025)
- [TextDiffuser-RL: Efficient and Robust Text Layout Optimization for High-Fidelity Text-to-Image Synthesis](https://mlsanity.com/paper/2505.19291/) (2025)
- [Tuning Just Enough: Lightweight Backdoor Attacks on Multi-Encoder Diffusion Models](https://mlsanity.com/paper/2603.04064/) (2026)
- [Personalizing Text-to-Image Generation to Individual Taste](https://mlsanity.com/paper/2604.07427/) (2026)
- [Divide-and-Denoise: A Game-Theoretic Method for Fairly Composing Diffusion Models](https://mlsanity.com/paper/2606.14756/) (2026)

## Abstract (arXiv)

Text-to-image diffusion models are typically trained to optimize the log-likelihood objective, which presents challenges in meeting specific requirements for downstream tasks, such as image aesthetics and image-text alignment. Recent research addresses this issue by refining the diffusion U-Net using human rewards through reinforcement learning or direct backpropagation. However, many of them overlook the importance of the text encoder, which is typically pretrained and fixed during training. In this paper, we demonstrate that by finetuning the text encoder through reinforcement learning, we can enhance the text-image alignment of the results, thereby improving the visual quality. Our primary motivation comes from the observation that the current text encoder is suboptimal, often requiring careful prompt adjustment. While fine-tuning the U-Net can partially improve performance, it remains suffering from the suboptimal text encoder. Therefore, we propose to use reinforcement learning with low-rank adaptation to finetune the text encoder based on task-specific rewards, referred as TexForce. We first show that finetuning the text encoder can improve the performance of diffusion models. Then, we illustrate that TexForce can be simply combined with existing U-Net finetuned models to get much better results without additional training. Finally, we showcase the adaptability of our method in diverse applications, including the generation of high-quality face and hand images.