mlsanityData to 7 Oct 2026

Moment Matching Q-Learning

arXiv 2605.29033 ↗First posted 27 May 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Score-based and flow-based generative models exhibit remarkable expressive capacity in capturing complex distributions, and have been extensively deployed in tasks ranging from image generation to reinforcement learning. Nevertheless, these models suffer from prolonged inference latency, which imposes a significant computational bottleneck in RL with iterative sampling. To overcome this limitation, we propose a new framework named Moment Matching Q-Learning (MoMa QL), which utilizes a technique from statistical hypothesis testing known as maximum mean discrepancy (MMD) that intend to match all orders of statistics between the original and target distribution. By enforcing strong regularization on all moment statistics, this algorithm guarantees distribution-level convergence for conditional score function and remains stable under various hyperparameters. Empirically, we show that our method MoMa QL is more computationally efficient with a comparable if not competitive performance in various D4RL tasks. Remarkably, by accelerating the action sampling process for flow-based policies, MoMa QL demonstrates superior performance in offline-to-online RL tasks because of faster and stronger adaptability for online interactive finetuning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 56 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

44 of 56 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learning6 Mar
  2. 2023Score Regularized Policy Optimization through Diffusion Behavior11 Oct
  3. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
  4. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  5. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  6. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 38

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  3. 2022Flow Matching for Generative Modelingcited by 5,426
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  9. 2023Consistency Modelscited by 1,778
  10. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  11. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  12. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
26 more
  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  2. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  6. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  7. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  9. 2021Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modelingcited by 594
  10. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  11. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  12. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  13. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  14. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  15. 2021Offline RL Without Off-Policy Evaluationcited by 150
  16. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  17. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  18. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  19. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  20. 2025Inductive Moment Matchingcited by 107
  21. 2014On the relation between optimal transport and Schr"odinger bridges: A stochastic control viewpointcited by 99
  22. 2023Guided Flows for Generative Modeling and Decision Makingcited by 99
  23. 2024FlowPolicy: Enabling Fast and Robust 3D Flow-based Policy via Consistency Flow Matching for Robot Manipulationcited by 99
  24. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  25. 2021Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learningcited by 26
  26. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RLcited by 21

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Flow Q-Learning
  2. 2026Score-Calibrated Flow for Sampling from Unnormalized Densities with Applications to Generative Online Reinforcement Learning
  3. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL
  4. 2026Q-learning with Adjoint Matching
  5. 2025One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow
  6. 2024Reward-Directed Score-Based Diffusion Models via q-Learning
  7. 2026Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies
  8. 2026GFlowRL: Scaling Distribution-Matching RL to Large Language Models