mlsanityData to 14 Oct 2026

Value Flows

arXiv 2510.07650 ↗First posted 9 Oct 2025cs.LG cs.AIcited by 15

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

While most reinforcement learning methods today flatten the distribution of future returns to a single scalar value, distributional RL methods exploit the return distribution to provide stronger learning signals and to enable applications in exploration and safe RL. While the predominant method for estimating the return distribution is by modeling it as a categorical distribution over discrete bins or estimating a finite number of quantiles, such approaches leave unanswered questions about the fine-grained structure of the return distribution and about how to distinguish states with high return uncertainty for decision-making. The key idea in this paper is to use modern, flexible flow-based models to estimate the full future return distributions and identify those states with high return variance. We do so by formulating a new flow-matching objective that generates probability density paths satisfying the distributional Bellman equation. Building upon the learned flow models, we estimate the return uncertainty of distinct states using a new flow derivative ODE. We additionally use this uncertainty information to prioritize learning a more accurate return estimation on certain transitions. We compare our method (Value Flows) with prior methods in the offline and online-to-online settings. Experiments on 37 state-based and 25 image-based benchmark tasks demonstrate that Value Flows achieves a 1.3× improvement on average in success rates. Website: https://pd-perry.github.io/value-flows Code: https://github.com/chongyi-zheng/value-flows

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 62 of its 85 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

62 of 85 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning11 Feb

Other references · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Flow Matching for Generative Modelingcited by 5,702
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  11. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  12. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
38 more
  1. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  4. 2015Prioritized Experience Replaycited by 1,099
  5. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  6. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  10. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  12. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  13. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  14. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  15. 2024One Step Diffusion via Shortcut Modelscited by 391
  16. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  17. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  18. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  19. 2024Flow Matching Guide and Codecited by 301
  20. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  21. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  22. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  23. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
  24. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  25. 2018Statistical Aspects of Wasserstein Distancescited by 203
  26. 2025Flow Q-Learningcited by 184
  27. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  28. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  29. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
  30. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  31. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  32. 2025Reinforcement Learning with Action Chunkingcited by 107
  33. 2024Foundation Policies with Hilbert Representationscited by 94
  34. 2021Conservative Offline Distributional Reinforcement Learningcited by 79
  35. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  36. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  37. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  38. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 63

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 6
  2. 2026Reinforcement Learning via Value Gradient Flowcited by 5
  3. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 5
  4. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learningcited by 2
  5. 2026Path-Coupled Bellman Flows for Distributional Reinforcement Learningcited by 2
  6. 2026Q-Learning With World Modelscited by 2
  7. 2026Quantile-Coupled Flow Matching for Distributional Reinforcement Learningcited by 1
  8. 2026Stochastic Actor-Critic: Mitigating Overestimation via Temporal Aleatoric Uncertaintycited by 0
  9. 2026Distributional Reinforcement Learning with Diffusion Bridge Criticscited by 0
  10. 2026Horizon Adaptive Offline Policy Learning via Value Stitchingcited by 0
  11. 2026VINE: Taming Generative Control Policies for Reinforcement Learningcited by 0
  12. 2026Retimed Bellman Flows: Escaping the Impossible Triangle of Velocity Bootstrappingcited by 0
3 more
  1. 2026From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learningcited by 0
  2. 2026Posterior-Averaged Bellman Flow Matching for Distributional Criticscited by 0
  3. 2026Occupancy Modeling via Drifting: One-Step Generation Enables Direct Policy Extractioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Parameter-Efficient Distributional RL via Normalizing Flows and a Geometry-Aware Cram'er Surrogate
  2. 2025Unleashing Flow Policies with Distributional Critics
  3. 2025Tractable Representations for Convergent Approximation of Distributional HJB Equations
  4. 2025Optimizing Return Distributions with Distributional Dynamic Programming
  5. 2025Distributional Reinforcement Learning on Path-dependent Options
  6. 2025Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation
  7. 2025FlowCritic: Bridging Value Estimation with Flow Matching in Reinforcement Learning
  8. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning