mlsanityData to 7 Oct 2026

Drift Q-Learning

arXiv 2606.00350 ↗First posted 29 May 2026cs.LG cs.AIcited by 2

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Offline reinforcement learning requires improving a policy from fixed data while avoiding out-of-distribution actions with unreliable value estimates. Diffusion and flow policies handle this trade-off by modeling the behavior distribution to regularize the RL objective, but they require iterative denoising, solver integrations, and in more efficient variants, distillation or other approximations at inference. We propose DriftQL, which combines a drift-based behavioral regularizer with critic-driven policy improvement. The value signal biases the policy toward high-value regions of the data support, while attraction and repulsion together keep generated actions near the data and prevent collapse onto a single mode. DriftQL is implemented as a single network with a unified training objective and generates actions in a single forward pass. On D4RL and OGBench, DriftQL consistently outperforms diffusion and flow methods, advancing the state of the art. Under degraded data quality, where the baselines visibly struggle, DriftQL remains close to its clean-data performance, positioning it as a promising alternative to diffusion and flow-based methods while maintaining the simplicity and efficiency of deterministic approaches. Project page: https://driftql.github.io/

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 36 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

32 of 36 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation4 Feb
  2. 2025One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learning19 Aug
  3. 2025Flow-Based Single-Step Completion for Efficient and Expressive Policy Learning26 Jun
  4. 2025FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning20 May
  5. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning7 Feb
  6. 2025Flow Q-Learning4 Feb
  7. 2024OGBench: Benchmarking Offline Goal-Conditioned RL26 Oct
  8. 2023Score Regularized Policy Optimization through Diffusion Behavior11 Oct
  9. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
  10. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  12. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 20

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  3. 2022Flow Matching for Generative Modelingcited by 5,426
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  11. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  12. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
8 more
  1. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  2. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  3. 2026Generative Modeling via Driftingcited by 98
  4. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  5. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  6. 2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
  7. 2025Safe Domain Randomization via Uncertainty-Aware Out-of-Distribution Detection and Policy Adaptationcited by 3
  8. 2026Contractive Diffusion Policies: Robust Action Diffusion via Contractive Score-Based Sampling with Differential Equationscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026FlowDPG: Deterministic Policy Gradient on Flow Matching Policies for Real-World Manipulationcited by 2
  2. 2026DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning
  2. 2026Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning
  3. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
  4. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning
  5. 2026Reinforcement Learning from Denoising Feedback
  6. 2023Boosting Continuous Control with Consistency Policy
  7. 2026Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning
  8. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning