mlsanityData to 14 Oct 2026

ExploreNet: Learning Where to Explore in Diffusion GRPO

arXiv 2609.38329 ↗First posted 29 Sep 2026cs.CVcited by 0

On the map: LLM reasoning → Diffusion reasoning and test-time scaling

Abstract (arXiv)

Group-relative RL methods such as Flow-GRPO post-train image generators by exploring with isotropic Gaussian noise added at every denoising step. This noise decides which rollouts the model learns from, yet it perturbs every channel and spatial position of the latent equally. In this paper, we instead show that latent elements differ in how much they change the generated image, so exploration should adapt to these differences. We introduce EXPLORENET to learn an adaptive exploration distribution. EXPLORENET is a policy that predicts a noise scale for every latent element from the current latent, the denoising step, and the prompt, before any reward is observed; it is trained on the reward spread of each rollout group and discarded after training, leaving inference unchanged. On Stable Diffusion 3.5 Medium, EXPLORENET improves held-out GenEval2 by 14% over Flow-GRPO, transfers to two independent compositional benchmarks and five preference and image-quality models, and reaches a 67.2% human preference win-rate. Overall, across our group-relative diffusion RL experiments, we find that exploration is learnable, the shape of the exploration distribution outweighs its magnitude, and rollout quality is more effective than rollout quantity.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion reasoning and test-time scaling, one of 17 directions in LLM reasoning. Builds on 50 of its 68 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

50 of 68 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models13 Aug

Other references · 49

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  5. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  6. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  7. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  8. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  9. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  10. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  11. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
  12. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
37 more
  1. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 450
  2. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 448
  3. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  4. 2025DanceGRPO: Unleashing GRPO on Visual Generationcited by 413
  5. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 288
  6. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 260
  7. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 178
  8. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 166
  9. 2024Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Controlcited by 143
  10. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 119
  11. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Modelscited by 97
  12. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 61
  13. 2025Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matchingcited by 56
  14. 2025GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluationcited by 52
  15. 2023Diffusion Models With Learned Adaptive Noisecited by 51
  16. 2024Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNetscited by 32
  17. 2025Personalized Preference Fine-tuning of Diffusion Modelscited by 22
  18. 2026DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignmentcited by 21
  19. 2024FIND: Fine-tuning Initial Noise Distribution with Policy Optimization for Diffusion Modelscited by 20
  20. 2026Manifold-Aware Exploration for Reinforcement Learning in Video Generationcited by 15
  21. 2025Reinforcing Diffusion Models by Direct Group Preference Optimizationcited by 14
  22. 2026E-GRPO: High Entropy Steps Drive Effective Reinforcement Learning for Flow Modelscited by 14
  23. 2025Smart-GRPO: Smartly Sampling Noise for Efficient RL of Flow-Matching Modelscited by 12
  24. 2025The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generationcited by 12
  25. 2026Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPOcited by 9
  26. 2025ImageReFL: Balancing Quality and Diversity in Human-Aligned Diffusion Modelscited by 8
  27. 2025Understanding Sampler Stochasticity in Training Diffusion Models for RLHFcited by 7
  28. 2026Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Versatile Image Generationcited by 5
  29. 2026Embedding-perturbed Exploration Preference Optimization for Flow Modelscited by 5
  30. 2025A Minimalist Method for Fine-tuning Text-to-Image Diffusion Modelscited by 3
  31. 2025Letting Trajectories Spread: Quality-Preserving Control for Diverse Flow Matchingcited by 3
  32. 2026Precise: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Modelscited by 3
  33. 2026NoiseRater: Meta-Learned Noise Valuation for Diffusion Model Trainingcited by 1
  34. 2026Learning Sampling Parameters for Diffusion Modelscited by 1
  35. 2026Explore or Converge? Stage-Guided Per-Step Optimization for Diffusion Modelscited by 1
  36. 2026Spatially Adaptive Noise Injectioncited by 1
  37. 2026Verifiable Visual Rewards Transfer from Synthetic Scenes to Natural Promptscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PAST: Prompt-Adaptive Sampling Termination for Efficient Diffusion Model
  2. 2026Enhanced Low-Density Region Exploration in Classifier-Guided Diffusion Models Through Modified Reverse Diffusion Sampling
  3. 2025CoCA: Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning
  4. 2026Critic-Guided Reinforcement Unlearning in Text-to-Image Diffusion
  5. 2026PreviewDiff: Multimodal Critic-Guided Search over Diffusion Latents
  6. 2026Learning To Sample From Diffusion Models Via Inverse Reinforcement Learning
  7. 2024Improving GFlowNets for Text-to-Image Diffusion Alignment
  8. 2025HiGS: History-Guided Sampling for Plug-and-Play Enhancement of Diffusion Models