mlsanityData to 7 Oct 2026

Retimed Bellman Flows: Escaping the Impossible Triangle of Velocity Bootstrapping

arXiv 2609.32828 ↗First posted 26 Sep 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Flow critics learn return distributions by transporting Gaussian noise to Bellman endpoints via continuous velocity fields. While velocity bootstrapping stabilizes training by querying a successor teacher, existing methods face a structural dilemma: on straight paths, no residual-free same-time affine mapping can preserve Gaussian initial noise while maintaining an unbiased target. To overcome this limitation, we introduce Retimed Bellman Flows (ReBF). ReBF queries the teacher critic at a dynamically shifted earlier flow time, aligning intermediate student and teacher trajectories. By combining this retimed clock with fresh, decoupled noise generation, ReBF constructs a provably conditionally unbiased velocity target that preserves the Bellman fixed point and contracts under Wasserstein distances. Empirically, ReBF reduces W1 distance to ground-truth return distributions by up to 7.7× on synthetic MRPs and outperforms existing flow critics across 38 challenging OGBench and D4RL offline reinforcement learning tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 30 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

21 of 30 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Path-Coupled Bellman Flows for Distributional Reinforcement Learning7 May
  2. 2025Unleashing Flow Policies with Distributional Critics27 Sep
  3. 2025Flow Q-Learning4 Feb
  4. 2024OGBench: Benchmarking Offline Goal-Conditioned RL26 Oct

Other references · 17

  1. 2022Flow Matching for Generative Modelingcited by 5,426
  2. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  5. 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 835
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  8. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  9. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  10. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 80
  11. 2021Conservative Offline Distributional Reinforcement Learningcited by 78
  12. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
5 more
  1. 2025Value Flowscited by 12
  2. 2024The surprising efficiency of temporal difference learning for rare event predictioncited by 7
  3. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
  4. 2024Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Spacecited by 4
  5. 2026Quantile-Coupled Flow Matching for Distributional Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
  2. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling
  3. 2026Score-Calibrated Flow for Sampling from Unnormalized Densities with Applications to Generative Online Reinforcement Learning
  4. 2026Optimal Self-Distillation for Rectified Flow via Linear Probing
  5. 2026Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning
  6. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning
  7. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies
  8. 2025Theoretical Guarantees for High Order Trajectory Refinement in Generative Flows