mlsanityData to 14 Oct 2026

DeFlow: Decoupling Manifold Modeling and Value Maximization for Offline Policy Extraction

arXiv 2601.10471 ↗First posted 15 Jan 2026cs.LGcited by 5

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

We present DeFlow, a decoupled offline RL framework that leverages flow matching to faithfully capture complex behavior manifolds. Optimizing generative policies is computationally prohibitive, typically necessitating backpropagation through ODE solvers. We address this by learning a lightweight refinement module within an explicit, data-derived trust region of the flow manifold, rather than sacrificing the iterative generation capability via single-step distillation. This way, we bypass solver differentiation and eliminate the need for balancing loss terms, ensuring stable improvement while fully preserving the flow's iterative expressivity. Empirically, DeFlow achieves superior performance on the challenging OGBench benchmark and demonstrates efficient offline-to-online adaptation.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 35 of its 39 references that are in the map. First posted on arXiv 15 Jan 2026.

Builds on

35 of 39 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL8 Sep
  2. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learning6 Mar
  3. 2023Score Regularized Policy Optimization through Diffusion Behavior11 Oct
  4. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
  5. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  6. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  7. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  8. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2022Flow Matching for Generative Modelingcited by 5,702
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  10. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  11. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  12. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
15 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  2. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  3. 2024Flow Matching Guide and Codecited by 301
  4. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  5. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  6. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  7. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 119
  8. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  9. 2023Synthetic Experience Replaycited by 101
  10. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 90
  11. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 78
  12. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 31
  13. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  14. 2025One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlowcited by 18
  15. 2025Dual Goal Representationscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Fisher Decorator: Refining Flow Policy via a Local Transport Mapcited by 2
  2. 2026Path-Space Mirror Descent for On-Policy Reinforcement Learning under the Generalized Schr"odinger Bridgecited by 0
  3. 2026Preserve Support, Not Correspondence: Dynamic Routing for Offline Reinforcement Learningcited by 0
  4. 2026CoDrift: Compositional Drifting for Offline Reinforcement Learningcited by 0
  5. 2026Diffusion Policy Improvement with Proposal-Conditioned Refinement Flowscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Flow-Based Single-Step Completion for Efficient and Expressive Policy Learning
  2. 2025Expressive Value Learning for Scalable Offline Reinforcement Learning
  3. 2026Decoupling Policy Extraction for Offline Reinforcement Learning
  4. 2026Multi-step Proximal Policy Improvement in Offline Reinforcement Learning
  5. 2026VGFM: Expressive Robot Policies via Dense Value Guidance in Flow Matching
  6. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling
  7. 2025Offline Reinforcement Learning with Generative Trajectory Policies
  8. 2025Flow-Based Policy for Online Reinforcement Learning