# Retimed Bellman Flows: Escaping the Impossible Triangle of Velocity Bootstrapping

arXiv: https://arxiv.org/abs/2609.32828 · HTML: https://mlsanity.com/paper/2609.32828/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Placed in

- Offline RL and imitation learning from data / [Diffusion and flow policies for offline RL](https://mlsanity.com/offlinerl/diffusion-and-flow-policies-for-offline-rl/)

## Builds on (most cited first, 21 of 21)

- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Improving and generalizing flow-based generative models with minibatch optimal transport](https://mlsanity.com/paper/2302.00482/) (2023)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [OGBench: Benchmarking Offline Goal-Conditioned RL](https://mlsanity.com/paper/2410.20092/) (2024)
- [Distributed Distributional Deterministic Policy Gradients](https://mlsanity.com/paper/1804.08617/) (2018)
- [Flow Q-Learning](https://mlsanity.com/paper/2502.02538/) (2025)
- [Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors](https://mlsanity.com/paper/2001.02811/) (2020)
- [Conservative Offline Distributional Reinforcement Learning](https://mlsanity.com/paper/2107.06106/) (2021)
- [floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL](https://mlsanity.com/paper/2509.06863/) (2025)
- [Value Flows](https://mlsanity.com/paper/2510.07650/) (2025)
- [Unleashing Flow Policies with Distributional Critics](https://mlsanity.com/paper/2509.23087/) (2025)
- [The surprising efficiency of temporal difference learning for rare event prediction](https://mlsanity.com/paper/2405.17638/) (2024)
- [Expressive Value Learning for Scalable Offline Reinforcement Learning](https://mlsanity.com/paper/2510.08218/) (2025)
- [Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space](https://mlsanity.com/paper/2410.01796/) (2024)
- [Path-Coupled Bellman Flows for Distributional Reinforcement Learning](https://mlsanity.com/paper/2605.08253/) (2026)
- [Quantile-Coupled Flow Matching for Distributional Reinforcement Learning](https://mlsanity.com/paper/2605.08515/) (2026)

## Abstract (arXiv)

Flow critics learn return distributions by transporting Gaussian noise to Bellman endpoints via continuous velocity fields. While velocity bootstrapping stabilizes training by querying a successor teacher, existing methods face a structural dilemma: on straight paths, no residual-free same-time affine mapping can preserve Gaussian initial noise while maintaining an unbiased target. To overcome this limitation, we introduce Retimed Bellman Flows (ReBF). ReBF queries the teacher critic at a dynamically shifted earlier flow time, aligning intermediate student and teacher trajectories. By combining this retimed clock with fresh, decoupled noise generation, ReBF constructs a provably conditionally unbiased velocity target that preserves the Bellman fixed point and contracts under Wasserstein distances. Empirically, ReBF reduces W1 distance to ground-truth return distributions by up to 7.7× on synthetic MRPs and outperforms existing flow critics across 38 challenging OGBench and D4RL offline reinforcement learning tasks.