mlsanityData to 7 Oct 2026

FlowCritic: Bridging Value Estimation with Flow Matching in Reinforcement Learning

arXiv 2510.22686 ↗First posted 26 Oct 2025cs.LG cs.AIcited by 4

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Reliable value estimation serves as the cornerstone of reinforcement learning (RL) by evaluating long-term returns and guiding policy improvement, significantly influencing the convergence speed and final performance. Existing works improve the reliability of value function estimation via multi-critic ensembles and distributional RL, yet the former merely combines multi point estimation without capturing distributional information, whereas the latter relies on discretization or quantile regression, limiting the expressiveness of complex value distributions. Inspired by flow matching's success in generative modeling, we propose a generative paradigm for value estimation, named FlowCritic. Departing from conventional regression for deterministic value prediction, FlowCritic leverages flow matching to model value distributions and generate samples for value estimation.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 32 of its 40 references that are in the map. First posted on arXiv 26 Oct 2025.

Builds on

32 of 40 references.

References · 32

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2022Flow Matching for Generative Modelingcited by 5,426
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  11. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  12. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
20 more
  1. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  2. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
  3. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  4. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  5. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  6. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 299
  7. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  8. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 281
  9. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  10. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  11. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  12. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  13. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  14. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  15. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 80
  16. 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 65
  17. 2023Distributional Soft Actor-Critic with Three Refinementscited by 22
  18. 2021Reward-Adaptive Reinforcement Learning: Dynamic Policy Gradient Optimization for Bipedal Locomotioncited by 12
  19. 2024RobotKeyframing: Learning Locomotion with High-Level Objectives via Mixture of Dense and Sparse Rewardscited by 12
  20. 2022How Does Return Distribution in Distributional Reinforcement Learning Help Optimization?cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026What Does Flow Matching Bring To TD Learning?cited by 2
  2. 2026ScoRe-Flow: Complete Distributional Control via Score-Based Reinforcement Learning for Flow Matchingcited by 1
  3. 2026Quantile-Coupled Flow Matching for Distributional Reinforcement Learningcited by 1
  4. 2026Distributional Reinforcement Learning with Diffusion Bridge Criticscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Training
  2. 2026VINE: Taming Generative Control Policies for Reinforcement Learning
  3. 2026Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learning
  4. 2026Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning
  5. 2026VGFM: Expressive Robot Policies via Dense Value Guidance in Flow Matching
  6. 2025Value Flows
  7. 2026Dual-Flow Reinforcement Learning with State-Aware Exploration
  8. 2025Unleashing Flow Policies with Distributional Critics