mlsanityData to 7 Oct 2026

Aligning Flow Map Policies with Optimal Q-Guidance

arXiv 2605.12416 ↗First posted 12 May 2026cs.LGcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Generative policies based on expressive model classes, such as diffusion and flow matching, are well-suited to complex control problems with highly multimodal action distributions. Their expressivity, however, comes at a significant inference cost: generating each action typically requires simulating many steps of the generative process, compounding latency across sequential decision-making rollouts. We introduce flow map policies, a novel class of generative policies designed for fast action generation by learning to take arbitrary-size jumps including one-step jumps-across the generative dynamics of existing flow-based policies. We instantiate flow map policies for offline-to-online reinforcement learning (RL) and formulate online adaptation as a trust-region optimization problem that improves the critic's Q-value while remaining close to the offline policy. We theoretically derive FLOW MAP Q-GUIDANCE (FMQ), a principled closed-form learning target that is optimal for adapting offline flow map policies under a critic-guided trust-region constraint. We further introduce Q-GUIDED BEAM SEARCH (QGBS), a stochastic flow-map sampler that combines renoising with beam search to enable iterative inference-time refinement. Across 12 challenging robotic manipulation and locomotion tasks from OGBench and RoboMimic, FMQ achieves state-of-the-art performance in offline-to-online RL, outperforming the previous one-step policy MVP by a relative improvement of 21.3% on the average success rate.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 47 of its 51 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

47 of 51 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026Diamond Maps: Efficient Reward Alignment via Stochastic Flow Maps5 Feb
  2. 2025One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learning19 Aug
  3. 2025Flow Q-Learning4 Feb
  4. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning30 May
  5. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning6 Feb
  6. 2023Diffusion Models for Reinforcement Learning: A Survey2 Nov
  7. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
  8. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  9. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
  10. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  12. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
2 more
  1. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  2. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 33

  1. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  3. 2022Flow Matching for Generative Modelingcited by 5,426
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
  12. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
21 more
  1. 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 835
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  3. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  4. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  6. 2025Mean Flows for One-step Generative Modelingcited by 524
  7. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  8. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  9. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  10. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  11. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  12. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  13. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  14. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  15. 2025Reinforcement Learning with Action Chunkingcited by 97
  16. 2025How to build a consistency model: Learning flow maps via self-distillationcited by 96
  17. 2023Non-Denoising Forward-Time Diffusionscited by 75
  18. 2026Meta Flow Maps enable scalable reward alignmentcited by 32
  19. 2024AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimizationcited by 19
  20. 2018Composable Action-Conditioned Predictors: Flexible Off-Policy Learning for Robot Navigationcited by 9
  21. 2026Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generationcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow
  2. 2026Flow Matching for Offline Reinforcement Learning with Discrete Actions
  3. 2026Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning
  4. 2026QPILOTS: Efficient Test-Time Q-Steering for Flow Policies
  5. 2026Discrete Flow Matching for Offline-to-Online Reinforcement Learning
  6. 2025Training Diffusion Policies via Prior-Mapping Co-Evolution
  7. 2026Refinement-based Flow Policy Optimization
  8. 2025Offline Reinforcement Learning with Generative Trajectory Policies