# From Competition to Synergy: Unlocking Reinforcement Learning for Subject-Driven Image Generation

arXiv: https://arxiv.org/abs/2510.18263 · HTML: https://mlsanity.com/paper/2510.18263/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [RL with diffusion and flow models](https://mlsanity.com/deeprl/rl-with-diffusion-and-flow-models/)

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025)
- [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](https://mlsanity.com/paper/2305.14720/) (2023)
- [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://mlsanity.com/paper/2303.11897/) (2023)
- [OminiControl: Minimal and Universal Control for Diffusion Transformer](https://mlsanity.com/paper/2411.15098/) (2024)
- [HPSv3: Towards Wide-Spectrum Human Preference Score](https://mlsanity.com/paper/2508.03789/) (2025)
- [VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation](https://mlsanity.com/paper/2312.14867/) (2023)
- [MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE](https://mlsanity.com/paper/2507.21802/) (2025)
- [T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoT](https://mlsanity.com/paper/2505.00703/) (2025)
- [Less-to-More Generalization: Unlocking More Controllability by In-Context Generation](https://mlsanity.com/paper/2504.02160/) (2025)
- [SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation](https://mlsanity.com/paper/2312.16272/) (2023)
- [DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation](https://mlsanity.com/paper/2406.16855/) (2024)
- [PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection](https://mlsanity.com/paper/2502.12119/) (2025)
- [GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learning](https://mlsanity.com/paper/2505.17022/) (2025)
- [Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator](https://mlsanity.com/paper/2411.15466/) (2024)
- [Generating Multi-Image Synthetic Data for Text-to-Image Customization](https://mlsanity.com/paper/2502.01720/) (2025)
- [Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment](https://mlsanity.com/paper/2502.14354/) (2025)
- [T2I-FactualBench: Benchmarking the Factuality of Text-to-Image Models with Knowledge-Intensive Concepts](https://mlsanity.com/paper/2412.04300/) (2024)
- [Flux Already Knows -- Activating Subject-Driven Image Generation without Training](https://mlsanity.com/paper/2504.11478/) (2025)
- [FlexIP: Dynamic Control of Preservation and Personality for Customized Image Generation](https://mlsanity.com/paper/2504.07405/) (2025)

## Built on by (most cited first, 4 of 4)

- [SenseNova-U1.5: Towards Native Unified Visual Intelligence](https://mlsanity.com/paper/2609.11929/) (2026)
- [Advances in GRPO for Generation Models: A Survey](https://mlsanity.com/paper/2603.06623/) (2026)
- [From Synthetic to Real: Toward Identity-Consistent Makeup Transfer with Synthetic and Real Data](https://mlsanity.com/paper/2605.07861/) (2026)
- [Squeezing Capacity from Multimodal Large Language Models for Subject-driven Generation](https://mlsanity.com/paper/2605.26111/) (2026)

## Abstract (arXiv)

Subject-driven image generation models face a fundamental trade-off between identity preservation (fidelity) and prompt adherence (editability). While online reinforcement learning (RL), specifically GPRO, offers a promising solution, we find that a naive application of GRPO leads to competitive degradation, as the simple linear aggregation of rewards with static weights causes conflicting gradient signals and a misalignment with the temporal dynamics of the diffusion process. To overcome these limitations, we propose Customized-GRPO, a novel framework featuring two key innovations: (i) Synergy-Aware Reward Shaping (SARS), a non-linear mechanism that explicitly penalizes conflicted reward signals and amplifies synergistic ones, providing a sharper and more decisive gradient. (ii) Time-Aware Dynamic Weighting (TDW), which aligns the optimization pressure with the model's temporal dynamics by prioritizing prompt-following in the early, identity preservation in the later. Extensive experiments demonstrate that our method significantly outperforms naive GRPO baselines, successfully mitigating competitive degradation. Our model achieves a superior balance, generating images that both preserve key identity features and accurately adhere to complex textual prompts.