mlsanityData to 7 Oct 2026

Reversal Q-Learning

arXiv 2606.17551 ↗First posted 16 Jun 2026cs.LG cs.AIcited by 2

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Iterative generative modeling techniques, such as flow matching, provide powerful tools to model complex behaviors for effective offline reinforcement learning (RL). In this work, we propose a new off-policy RL algorithm that trains a flow policy based on prior data. Our idea starts from the "expanded" Markov decision process (MDP) framework, which treats individual flow refinement steps as separate actions in an MDP. To enable off-policy RL within this framework, we apply two techniques: we generate virtual on-policy trajectories (by "reversing" flows) to make this framework compatible with prior data, and we apply a bias-and-variance reduction technique to mitigate the curse of horizon in off-policy RL. We call the resulting algorithm Reversal Q-learning (RQL). RQL has several advantages over previous flow-based RL methods: it does not suffer from backpropagation through time, makes better use of the learned value function, and directly trains the full, expressive flow policy. Through our experiments on 50 challenging simulated robotic tasks, we show that RQL leads to the best average offline RL performance compared to state-of-the-art flow-based offline RL algorithms.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 57 of its 65 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

57 of 65 references.

Closest ancestors · 22

Same direction, newest first

  1. 2026Q-learning with Adjoint Matching20 Jan
  2. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL8 Sep
  3. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learning18 Jun
  4. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learning6 Mar
  5. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning7 Feb
  6. 2025Flow Q-Learning4 Feb
  7. 2024Diffusion Policy Policy Optimization1 Sep
  8. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  9. 2024Policy-Guided Diffusion9 Apr
  10. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning6 Feb
  11. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning5 Feb
  12. 2024Simple Hierarchical Planning with Diffusion5 Jan
10 more
  1. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  2. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
  3. 2023Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning10 Jul
  4. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  5. 2023Training Diffusion Models with Reinforcement Learning22 May
  6. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
  7. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  8. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  9. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  10. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 35

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2022Flow Matching for Generative Modelingcited by 5,426
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  8. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  11. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  12. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
23 more
  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  2. 2022Building Normalizing Flows with Stochastic Interpolantscited by 964
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  4. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  6. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  7. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  8. 2024Flow Matching Guide and Codecited by 291
  9. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  10. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  11. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  12. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  13. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  14. 2023Synthetic Experience Replaycited by 100
  15. 2023Guided Flows for Generative Modeling and Decision Makingcited by 99
  16. 2025Reinforcement Learning with Action Chunkingcited by 97
  17. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  18. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  19. 2025Horizon Reduction Makes RL Scalablecited by 55
  20. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  21. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  22. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 27
  23. 2024AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimizationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Diffusion Policy Improvement with Proposal-Conditioned Refinement Flowscited by 0
  2. 2026Q-Learning with Scalar Adjoint Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Flow Matching for Offline Reinforcement Learning with Discrete Actions
  2. 2026Discrete Flow Matching for Offline-to-Online Reinforcement Learning
  3. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
  4. 2026Causal Flow Q-Learning for Robust Offline Reinforcement Learning
  5. 2026Aligning Flow Map Policies with Optimal Q-Guidance
  6. 2026Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning
  7. 2023Projected Off-Policy Q-Learning (POP-QL) for Stabilizing Offline Reinforcement Learning
  8. 2021Offline RL Without Off-Policy Evaluation