QAMM: Adjoint MeanFlow Matching for Few-Step Offline Reinforcement Learning
On the map: Diffusion models → Diffusion and flow policies for robotics
also in 1 more
Abstract (arXiv)
Flow policies can model rich action distributions, but their iterative sampling limits decision speed. Adjoint matching uses the critic's action gradient to improve a flow policy without backpropagating through its sampling trajectory, yet its supervision is defined for instantaneous velocities. We propose QAMM, a method that turns the critic-derived adjoint signal into supervision for MeanFlow's average velocity. The resulting policy learns finite-interval transport directly and generates actions with few network evaluations. We derive the adjoint MeanFlow target, specify its gradient boundaries, and train it with an offline actor-critic. On ten HumanoidMaze tasks, QAMM produces effective two-call policies and achieves competitive performance against strong flow-policy baselines. These results show that adjoint-based Q optimization can be combined with average-velocity learning to obtain expressive offline policies with few-step action generation.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 23 of its 26 references that are in the map. First posted on arXiv 28 Sep 2026.
Builds on
23 of 26 references.
Closest ancestors · 12
Same direction, newest first
- 2025One-Step Flow Policy Mirror Descent31 Jul
- 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learning18 Jun
- 2025FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning20 May
- 2025Energy-Weighted Flow Matching for Offline Reinforcement Learning6 Mar
- 2025Flow Q-Learning4 Feb
- 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control13 Sep
- 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning31 May
- 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
- 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
- 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
- 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
- 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug
Other references · 11
- 2022Flow Matching for Generative Modelingcited by 5,702
- 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
- 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
- 2025Mean Flows for One-step Generative Modelingcited by 561
- 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
- 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
- 2025Improved Mean Flows: On the Challenges of Fastforward Generative Modelscited by 110
- 2025Much Ado About Noising: Dispelling the Myths of Generative Robotic Controlcited by 36
- 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
- 2026MeanFlowNFT: Bringing Forward-Process RL to Average-Velocity Generatorscited by 5
- 2026Trust Region Q Adjoint Matchingcited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Q-Learning with Scalar Adjoint Matching
- 2026Q-learning with Adjoint Matching
- 2026Entropy-Regularized Adjoint Matching for Offline Reinforcement Learning
- 2026Score-Based One-step MeanFlow Policy Optimization
- 2026QF3: Fast Flow RL with Filtered Q-Gradients
- 2026Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipeline
- 2026Flow Matching for Offline Reinforcement Learning with Discrete Actions
- 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning