mlsanityData to 7 Oct 2026

SuperFlow: Training Flow Matching Models with RL on the Fly

arXiv 2512.17951 ↗First posted 17 Dec 2025cs.CVcited by 10

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Recent progress in flow-based generative models and reinforcement learning (RL) has improved text-image alignment and visual quality. However, current RL training for flow models still has two main problems: (i) GRPO-style fixed per-prompt group sizes ignore variation in sampling importance across prompts, which leads to inefficient sampling and slower training; and (ii) trajectory-level advantages are reused as per-step estimates, which biases credit assignment along the flow. We propose SuperFlow, an RL training framework for flow-based models that adjusts group sizes with variance-aware sampling and computes step-level advantages in a way that is consistent with continuous-time flow dynamics. Empirically, SuperFlow reaches promising performance while using only 5.4% to 56.3% of the original training steps and reduces training time by 5.2% to 16.7% without any architectural changes. On standard text-to-image (T2I) tasks, including text rendering, compositional image generation, and human preference alignment, SuperFlow improves over SD3.5-M by 4.6% to 47.2%, and over Flow-GRPO by 1.7% to 16.0%.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 63 of its 77 references that are in the map. First posted on arXiv 17 Dec 2025.

Builds on

63 of 77 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025TempFlow-GRPO: When Timing Matters for GRPO in Flow Models6 Aug
  2. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE29 Jul
  3. 2025Towards Self-Improvement of Diffusion Models via Group Preference Optimization16 May
  4. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  5. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  6. 2025Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization3 Feb
  7. 2024Diffusion Policy Policy Optimization1 Sep
  8. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  9. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  10. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models25 May
  11. 2023Training Diffusion Models with Reinforcement Learning22 May

Other references · 52

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2022Flow Matching for Generative Modelingcited by 5,426
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  9. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  10. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  11. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,375
  12. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
38 more
  1. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  2. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  3. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  4. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  5. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  6. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  7. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  8. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  9. 2025Mean Flows for One-step Generative Modelingcited by 524
  10. 2025Process Reinforcement through Implicit Rewardscited by 478
  11. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  12. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
  13. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  14. 2025Transfer between Modalities with MetaQueriescited by 231
  15. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  16. 2024Free Process Rewards without Process Labelscited by 162
  17. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  18. 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
  19. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  20. 2024Golden Noise for Diffusion Models: A Learning Frameworkcited by 91
  21. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  22. 2025GPT-IMAGE-EDIT-1.5M: A Million-Scale, GPT-Generated Image Datasetcited by 68
  23. 2025Diffuse and Disperse: Image Generation with Representation Regularizationcited by 67
  24. 2025Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Modelcited by 59
  25. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 59
  26. 2025BLIP3o-NEXT: Next Frontier of Native Image Generationcited by 50
  27. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 48
  28. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shapingcited by 48
  29. 2025Fractal Generative Modelscited by 44
  30. 2025Is Noise Conditioning Necessary for Denoising Generative Models?cited by 42
  31. 2025ImageGen-CoT: Enhancing Text-to-Image In-context Learning with Chain-of-Thought Reasoningcited by 37
  32. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  33. 2025R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generationcited by 31
  34. 2025Test-Time Scaling of Diffusion Models via Noise Trajectory Searchcited by 28
  35. 2025Single-stream Policy Optimizationcited by 27
  36. 2025Denoising Hamiltonian Network for Physical Reasoningcited by 10
  37. 2025UniGen-1.5: Enhancing Image Generation and Editing through Reward Unification in Reinforcement Learningcited by 8
  38. 2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPOcited by 6
  2. 2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 3
  3. 2026Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shapingcited by 3
  4. 2026When Rubrics Fail: Error Enumeration as Reward in Reference-Free RL Post-Training for Virtual Try-Oncited by 1
  5. 2026Advances in GRPO for Generation Models: A Surveycited by 1
  6. 2026Adaptive Multi-Stage Patent Claim Generation with Unified Quality Assessmentcited by 0
  7. 2026ProHiFlo: Hierarchical Flow Matching with Functional Guidance for De Novo Protein Generationcited by 0
  8. 2026FinInvest-GTCN: Explainable Graph-Temporal-Causal Modeling for Risk-Aware Investment Decision Optimizationcited by 0
  9. 2026EVLA: An Electro-Aware Multimodal Assistant for Physically-Grounded Driving Reasoning and Controlcited by 0
  10. 2026A3M: Adaptive, Adversarial and Multi-Objective Learning for Strategic Bidding in Repeated Auctionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Faster and Better Alignment for Flow Matching Models via Step-aware Advantages
  2. 2026PromptRL: Prompt Matters in RL for Flow-Based Image Generation
  3. 2025Smart-GRPO: Smartly Sampling Noise for Efficient RL of Flow-Matching Models
  4. 2025Fine-tuning Flow Matching Generative Models with Intermediate Feedback
  5. 2026Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Models
  6. 2025Principled RL for Flow Matching Emerges from the Chunk-level Policy Optimization
  7. 2026LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectories
  8. 2025Value Gradient Guidance for Flow Matching Alignment