mlsanityData to 14 Oct 2026

Backpropagating Verifier Rewards Through Continuous Flow Language Models

arXiv 2610.19080 ↗First posted 13 Oct 2026cs.LGcited by 0

On the map: Diffusion models → Discrete and text diffusion also fits: Diffusion and flow policies for robotics

Abstract (arXiv)

Reinforcement learning (RL) has substantially improved the reasoning of autoregressive and discrete-diffusion language models. Language models that generate continuous token embeddings are emerging as a strong alternative, but RL post-training for them remains understudied. We propose method, a post-training framework for continuous flow language models. Such a model denoises token embeddings over multiple steps and decodes them into discrete tokens only at the last step. To obtain pathwise gradients, which backpropagate through continuous actions and have much lower variance than REINFORCE, we introduce a critic that estimates the value of intermediate token embeddings, bypassing the non-differentiable decoding step. The critic must also keep track of the value function as the policy changes; to this end, we design a critic that predicts a value from every token representation and improves policy optimization. Across four reasoning tasks, method reliably improves a continuous flow language model through RL post-training and outperforms baseline methods of the same type.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 42 of its 48 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

42 of 48 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026ELF: Embedded Language Flows11 May
  2. 2025Large Language Diffusion Models14 Feb
  3. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  4. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  5. 2022Continuous diffusion for categorical data28 Nov
  6. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2022Flow Matching for Generative Modelingcited by 5,702
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  8. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  9. 2015Continuous control with deep reinforcement learningcited by 3,407
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  12. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
24 more
  1. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  2. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  3. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 939
  4. 2025Group Sequence Policy Optimizationcited by 835
  5. 2022Large Language Models Can Self-Improvecited by 664
  6. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
  7. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 450
  8. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  9. 2025DanceGRPO: Unleashing GRPO on Visual Generationcited by 413
  10. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 373
  11. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  12. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  13. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  14. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
  15. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 178
  16. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  17. 2024Stream of Search (SoS): Learning to Search in Languagecited by 145
  18. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  19. 2025Flow Matching Policy Gradientscited by 99
  20. 2024Causal Language Modeling Can Elicit Search and Reasoning Capabilities on Logic Puzzlescited by 38
  21. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friendscited by 15
  22. 2025Diffusion Alignment as Variational Expectation-Maximizationcited by 4
  23. 2026Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Modelscited by 4
  24. 2026Self-Consistent Flow: Unifying Velocity and Endpoint Prediction for Rectified Flow Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning with Promising Tokens for Large Language Models
  2. 2026Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
  3. 2026PACT: From Credit Assignment to Critic Alignment
  4. 2026Not only where, But when: Temporal Scheduling for RLVR
  5. 2026Not all tokens are needed(NAT): token efficient reinforcement learning
  6. 2025How Reinforcement Learning After Next-Token Prediction Facilitates Learning
  7. 2025NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learning
  8. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions