# Self-Reflective Reinforcement Learning for Diffusion-based Image Reasoning Generation

arXiv: https://arxiv.org/abs/2505.22407 · HTML: https://mlsanity.com/paper/2505.22407/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 25 of its 56 references that are in the map.
- First posted on arXiv 28 May 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Diffusion models / [Discrete and text diffusion](https://mlsanity.com/diffusion/discrete-and-text-diffusion/)

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2503.06749/) (2025)
- [R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization](https://mlsanity.com/paper/2503.10615/) (2025)
- [MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning](https://mlsanity.com/paper/2503.07365/) (2025)
- [Diffusion Policy Policy Optimization](https://mlsanity.com/paper/2409.00588/) (2024)
- [T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoT](https://mlsanity.com/paper/2505.00703/) (2025)
- [CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models](https://mlsanity.com/paper/2406.08070/) (2024)
- [Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step](https://mlsanity.com/paper/2501.13926/) (2025)
- [MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency](https://mlsanity.com/paper/2502.09621/) (2025)
- [MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine](https://mlsanity.com/paper/2407.08739/) (2024)
- [Classifier-Free Guidance is a Predictor-Corrector](https://mlsanity.com/paper/2408.09000/) (2024)
- [Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards](https://mlsanity.com/paper/2503.11240/) (2025)
- [Self-Guidance: Boosting Flow and Diffusion Generation on Their Own](https://mlsanity.com/paper/2412.05827/) (2024)

## Built on by (most cited first, 6 of 6)

- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [A Survey on Parallel Reasoning](https://mlsanity.com/paper/2510.12164/) (2025)
- [MILR: Improving Multimodal Image Generation via Test-Time Latent Reasoning](https://mlsanity.com/paper/2509.22761/) (2025)
- [SciIR: A Large-scale Training Dataset and Benchmark for Scientific Image Reasoning Generation](https://mlsanity.com/paper/2606.30124/) (2026)
- [PhySe-RPO: Physics and Semantics Guided Relative Policy Optimization for Diffusion-Based Surgical Smoke Removal](https://mlsanity.com/paper/2603.22844/) (2026)
- [Uni-Synergy: Bridging Understanding and Generation for Personalized Reasoning via Co-operative Reinforcement Learning](https://mlsanity.com/paper/2605.10445/) (2026)

## Abstract (arXiv)

Diffusion models have recently demonstrated exceptional performance in image generation task. However, existing image generation methods still significantly suffer from the dilemma of image reasoning, especially in logic-centered image generation tasks. Inspired by the success of Chain of Thought (CoT) and Reinforcement Learning (RL) in LLMs, we propose SRRL, a self-reflective RL algorithm for diffusion models to achieve reasoning generation of logical images by performing reflection and iteration across generation trajectories. The intermediate samples in the denoising process carry noise, making accurate reward evaluation difficult. To address this challenge, SRRL treats the entire denoising trajectory as a CoT step with multi-round reflective denoising process and introduces condition guided forward process, which allows for reflective iteration between CoT steps. Through SRRL-based iterative diffusion training, we introduce image reasoning through CoT into generation tasks adhering to physical laws and unconventional physical phenomena for the first time. Notably, experimental results of case study exhibit that the superior performance of our SRRL algorithm even compared with GPT-4o. The project page is https://jadenpan0.github.io/srrl.github.io/.