# Can We Perform Online RL for Image Editing without Editing Rewards?

arXiv: https://arxiv.org/abs/2608.22780 · HTML: https://mlsanity.com/paper/2608.22780/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 24 Aug 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)
- Deep reinforcement learning methods / [RL with diffusion and flow models](https://mlsanity.com/deeprl/rl-with-diffusion-and-flow-models/)

## Builds on (most cited first, 21 of 21)

- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025)
- [Step1X-Edit: A Practical Framework for General Image Editing](https://mlsanity.com/paper/2504.17761/) (2025)
- [DanceGRPO: Unleashing GRPO on Visual Generation](https://mlsanity.com/paper/2505.07818/) (2025)
- [Human Preference Score: Better Aligning Text-to-Image Models with Human Preference](https://mlsanity.com/paper/2303.14420/) (2023)
- [Diffusion Model-Based Image Editing: A Survey](https://mlsanity.com/paper/2402.17525/) (2024)
- [GenAI Arena: An Open Evaluation Platform for Generative Models](https://mlsanity.com/paper/2406.04485/) (2024)
- [Advantage Weighted Matching: Aligning RL with Pretraining in Diffusion Models](https://mlsanity.com/paper/2509.25050/) (2025)
- [EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modeling](https://mlsanity.com/paper/2509.23909/) (2025)
- [EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing](https://mlsanity.com/paper/2509.26346/) (2025)
- [TextPecker: Rewarding Structural Anomaly Quantification for Enhancing Visual Text Rendering](https://mlsanity.com/paper/2602.20903/) (2026)
- [UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation](https://mlsanity.com/paper/2603.23500/) (2026)
- [RealGen: Photorealistic Text-to-Image Generation via Detector-Guided Rewards](https://mlsanity.com/paper/2512.00473/) (2025)
- [Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation](https://mlsanity.com/paper/2603.12247/) (2026)
- [Leveraging Verifier-Based Reinforcement Learning in Image Editing](https://mlsanity.com/paper/2604.27505/) (2026)
- [SpatialReward: Bridging the Perception Gap in Online RL for Image Editing via Explicit Spatial Reasoning](https://mlsanity.com/paper/2602.07458/) (2026)
- [Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?](https://mlsanity.com/paper/2605.15855/) (2026)
- [Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Survey](https://mlsanity.com/paper/2505.17352/) (2025)

## Built on by (most cited first, 1 of 1)

- [SwiftExplorer: Training-free Diffusion Model Alignment with Swift Diversity Exploration](https://mlsanity.com/paper/2609.06651/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) enables direct preference optimization for image editing through editing-specific rewards, which remain less developed due to costly triplet supervision and complex task-dependent calibration. In contrast, text-to-image (T2I) generation benefits from a mature and diverse reward ecosystem spanning semantic alignment, aesthetics, realism, glyph shape, and other visual preferences. Extending this ecosystem to image editing would substantially broaden the range of visual preferences accessible to RL-based optimization, prompting the central question: Can We Perform Image Editing RL without Editing Rewards? In this paper, we argue that the standard image editing dimensions have potential to be mapped to the T2I reward space: image quality can transfer directly, prompt following can be aligned through a description of the desired visual state, and reference consistency admits a coarse semantic conversion by encoding the source content to preserve. However, editing instructions specify relative changes, whereas T2I rewards require self-contained target descriptions; moreover, semantically valid captions from generic vision-language models may be incompatible with the frozen reward. Hence, we further introduce Lever-Edit, a two-stage framework that learns a reward-aligned captioner for counterfactual target descriptions, freezes it, and optimizes the editing policy solely with the transferred T2I reward. Experiments show competitive editing alignment and source preservation against editing-reward-based fine-tuning, while outperforming intuitive transfer baselines.