mlsanityData to 7 Oct 2026

Path-Coupled Bellman Flows for Distributional Reinforcement Learning

arXiv 2605.08253 ↗First posted 7 May 2026cs.LG cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Distributional reinforcement learning (DRL) models the full return distribution, but existing finite-support or quantile-based methods rely on projections, while recent flow-based approaches can suffer from boundary mismatch at the flow source or from high-variance bootstrapping when current and successor noises are independent. We propose Path-Coupled Bellman Flows (PCBF), a continuous-time DRL method that learns return distributions with flow matching using source-consistent Bellman-coupled paths: the current path starts from the required base prior at t=0, reaches the Bellman target at t=1, and maintains a pathwise affine relation to the successor flow at intermediate times (without requiring time-t marginals to satisfy a distributional Bellman fixed point for all t). PCBF couples current and successor return flows through shared base noise and uses a λ-parameterized control-variate target: λ=0 recovers an unbiased sample Bellman target, while λ>0 trades controlled bias for variance reduction. Experiments on analytically tractable MRPs, OGBench, and D4RL show improved distributional fidelity and training stability, and competitive offline RL performance.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 37 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

29 of 37 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Flow Matching for Offline Reinforcement Learning with Discrete Actions5 Feb
  2. 2025Unleashing Flow Policies with Distributional Critics27 Sep
  3. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learning6 Mar
  4. 2025Flow Q-Learning4 Feb
  5. 2024OGBench: Benchmarking Offline Goal-Conditioned RL26 Oct
  6. 2023Score Regularized Policy Optimization through Diffusion Behavior11 Oct
  7. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  8. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 21

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  3. 2022Flow Matching for Generative Modelingcited by 5,426
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  10. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  11. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  12. 2024Flow Matching Guide and Codecited by 291
9 more
  1. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  2. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 80
  3. 2021Conservative Offline Distributional Reinforcement Learningcited by 78
  4. 2025Temporal Difference Flowscited by 20
  5. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  6. 2025Value Flowscited by 12
  7. 2024The surprising efficiency of temporal difference learning for rare event predictioncited by 7
  8. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
  9. 2024Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Spacecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Retimed Bellman Flows: Escaping the Impossible Triangle of Velocity Bootstrappingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Quantile-Coupled Flow Matching for Distributional Reinforcement Learning
  2. 2024Bellman Diffusion Models
  3. 2026Flow-based Policy With Distributional Reinforcement Learning in Trajectory Optimization
  4. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies
  5. 2023A Coupled Flow Approach to Imitation Learning
  6. 2026Refinement-based Flow Policy Optimization
  7. 2026GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policies
  8. 2024The Role of Inherent Bellman Error in Offline Reinforcement Learning with Linear Function Approximation