# VideoAWR: Reinforcing Video Flow Models with Decoupled-Timestep Advantage-Weighted Rectification

arXiv: https://arxiv.org/abs/2610.13338 · HTML: https://mlsanity.com/paper/2610.13338/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 27 of 27)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025)
- [DanceGRPO: Unleashing GRPO on Visual Generation](https://mlsanity.com/paper/2505.07818/) (2025)
- [Improving Video Generation with Human Feedback](https://mlsanity.com/paper/2501.13918/) (2025)
- [DiffusionNFT: Online Diffusion Reinforcement with Forward Process](https://mlsanity.com/paper/2509.16117/) (2025)
- [HPSv3: Towards Wide-Spectrum Human Preference Score](https://mlsanity.com/paper/2508.03789/) (2025)
- [MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE](https://mlsanity.com/paper/2507.21802/) (2025)
- [VideoDPO: Omni-Preference Alignment for Video Diffusion Generation](https://mlsanity.com/paper/2412.14167/) (2024)
- [Video Diffusion Alignment via Reward Gradients](https://mlsanity.com/paper/2407.08737/) (2024)
- [TempFlow-GRPO: When Timing Matters for GRPO in Flow Models](https://mlsanity.com/paper/2508.04324/) (2025)
- [LongCat-Video Technical Report](https://mlsanity.com/paper/2510.22200/) (2025)
- [Diffusion Guidance Is a Controllable Policy Improvement Operator](https://mlsanity.com/paper/2505.23458/) (2025)
- [OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimization](https://mlsanity.com/paper/2412.15159/) (2024)
- [Diffusion Models without Classifier-free Guidance](https://mlsanity.com/paper/2502.12154/) (2025)
- [FlowAWR: Online Adaptive Flow Reinforcement via Advantage-Weighted Rectification](https://mlsanity.com/paper/2606.30376/) (2026)
- [Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization](https://mlsanity.com/paper/2605.15980/) (2026)

## Abstract (arXiv)

Regression-based RL for flow models mandates sampling without classifier-free guidance (CFG) to align the rectification target with the reference marginal velocity. Extending Advantage-Weighted Rectification (AWR), which dynamically scales this regression target based on reward, to video generation encounters a cold-start pathology: unguided rollouts collapse into low-fidelity regions where intra-group reward gaps remain wide but no longer track perceptual quality, rendering the optimization target arbitrary. Conversely, utilizing guided rollouts to restore fidelity, while keeping the regression target tied to the guided sampling distribution, provably triggers a geometric guidance blow-up under exponential-moving-average (EMA) updates. We propose VideoAWR, which resolves this tension by untying sampling from the optimization anchor. An EMA reference policy generates guided rollouts to restore reward separability, while the trainable policy is rectified against the unguided conditional anchor. We prove this decoupled objective is mathematically exact: the centered advantage cancels the anchor in expectation, filtering out the reward-neutral guidance offset and retaining only the reward-correlated covariance increment. Consequently, inference runs single-branch and CFG-free, yet inference-time CFG preserves its original semantics. Furthermore, we introduce Decoupled Timestep Rectification, supervising each rollout at an independently sampled timestep without bias to improve sample efficiency. Experiments on Wan2.1-1.3B and 14B demonstrate stable optimization and superior visual and motion quality in both CFG-free and CFG-enabled regimes. Code is available at pdfstartlink attr/Border [0 0 0] user/Subtype /Link /A << /S /URI /URI (https://github.com/Fuzm7/FlowAWR) >>% textcolorpearDarkVideoAWRpdfendlink.