mlsanityData to 7 Oct 2026

Q-learning with Adjoint Matching

arXiv 2601.14234 ↗First posted 20 Jan 2026cs.LG cs.AI cs.ROcited by 33

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

We propose Q-learning with Adjoint Matching (QAM), a novel TD-based reinforcement learning (RL) algorithm that tackles a long-standing challenge in continuous-action RL: efficient optimization of an expressive diffusion or flow-matching policy with respect to a parameterized Q-function. Effective optimization requires exploiting the first-order information of the critic, but it is challenging to do so for flow or diffusion policies because direct gradient-based optimization via backpropagation through their multi-step denoising process is numerically unstable. Existing methods work around this either by only using the value and discarding the gradient information, or by relying on approximations that sacrifice policy expressivity or bias the learned policy. QAM sidesteps both of these challenges by leveraging adjoint matching, a recently proposed technique in generative modeling, which transforms the critic's action gradient to form a step-wise objective function that is free from unstable backpropagation, while providing an unbiased, expressive policy at the optimum. Combined with temporal-difference backup for critic learning, QAM consistently outperforms prior approaches on hard, sparse reward tasks in both offline and offline-to-online RL.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 72 of its 77 references that are in the map. First posted on arXiv 20 Jan 2026.

Builds on

72 of 77 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025One-Step Flow Policy Mirror Descent31 Jul
  2. 2025Flow Matching Policy Gradients28 Jul
  3. 2025Flow-Based Single-Step Completion for Efficient and Expressive Policy Learning26 Jun
  4. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learning18 Jun
  5. 2025Energy-Weighted Flow Matching for Offline Reinforcement Learning6 Mar
  6. 2025Flow Q-Learning4 Feb
  7. 2025Efficient Online Reinforcement Learning for Diffusion Policy1 Feb
  8. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Control13 Sep
  9. 2024Diffusion Policy Policy Optimization1 Sep
  10. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient2 Jun
  11. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning31 May
  12. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning30 May
11 more
  1. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization25 May
  2. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning6 Feb
  3. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  4. 2023Score Regularized Policy Optimization through Diffusion Behavior11 Oct
  5. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning9 Oct
  6. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
  7. 2023Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning10 Jul
  8. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  9. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  10. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  11. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 49

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  8. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  10. 2022Video Diffusion Modelscited by 2,280
  11. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  12. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
37 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  2. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  4. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  5. 2024Flow Matching Guide and Codecited by 291
  6. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  7. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 216
  8. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  9. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  10. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 201
  11. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  12. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
  13. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  14. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  15. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  16. 2025Reinforcement Learning with Action Chunkingcited by 97
  17. 2025Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Expertscited by 85
  18. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  19. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
  20. 2024Classifier-Free Guidance is a Predictor-Correctorcited by 79
  21. 2025Adjoint Sampling: Highly Scalable Diffusion Samplers via Adjoint Matchingcited by 78
  22. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  23. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  24. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
  25. 2024Particle Denoising Diffusion Samplercited by 63
  26. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
  27. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  28. 2025Horizon Reduction Makes RL Scalablecited by 55
  29. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  30. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  31. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learningcited by 40
  32. 2023Stochastic Optimal Control Matchingcited by 39
  33. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  34. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  35. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  36. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representationscited by 20
  37. 2024Leveraging Skills from Unlabeled Prior Data for Efficient Online Explorationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 33.

33 papers.

Sort:

By area:

  1. 2026Flow Policy Gradients for Robot Controlcited by 14
  2. 2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policiescited by 12
  3. 2026EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Modelscited by 7
  4. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 5
  5. 2026Adjoint Matching through the Lens of the Stochastic Maximum Principle in Optimal Controlcited by 4
  6. 2026Reinforcement Learning via Value Gradient Flowcited by 4
  7. 2026FASTER: Value-Guided Sampling for Fast RLcited by 4
  8. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 4
  9. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 4
  10. 2026Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inferencecited by 3
  11. 2026Trust Region Q Adjoint Matchingcited by 3
  12. 2026Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policiescited by 2
21 more
  1. 2026QPILOTS: Efficient Test-Time Q-Steering for Flow Policiescited by 2
  2. 2026Reversal Q-Learningcited by 2
  3. 2026FlowDPG: Deterministic Policy Gradient on Flow Matching Policies for Real-World Manipulationcited by 2
  4. 2026Q-Learning With World Modelscited by 2
  5. 2026Decoupling Policy Extraction for Offline Reinforcement Learningcited by 2
  6. 2026Entropy-Regularized Adjoint Matching for Offline Reinforcement Learningcited by 1
  7. 2026Drifting Field Policy: A One-Step Generative Policy via Wasserstein Gradient Flowcited by 1
  8. 2026StructRL: Structured Action-Space Exploration for Flow-Based VLAscited by 1
  9. 2026ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Networkcited by 0
  10. 2026The Diffusion Encodercited by 0
  11. 2026Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimizationcited by 0
  12. 2026Reinforcement Learning for Flow-Matching Policies with Density Transportcited by 0
  13. 2026DiPOD: Diffusion Policy Optimization without Drifting Apartcited by 0
  14. 2026Horizon Adaptive Offline Policy Learning via Value Stitchingcited by 0
  15. 2026Optimal Transport Q-Learning for Flow Policy Steering and Accelerationcited by 0
  16. 2026VINE: Taming Generative Control Policies for Reinforcement Learningcited by 0
  17. 2026Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuningcited by 0
  18. 2026Reinforcement Learning for Real-Time Vision-Language-Action Policiescited by 0
  19. 2026G2MAF: Test-Time Gradient Guidance for Multi-Agent Flow Policiescited by 0
  20. 2026Adjoint Guidance Flow: Amortized Critic Guidance for VLA Policiescited by 0
  21. 2026Diffusion Policy Improvement with Proposal-Conditioned Refinement Flowscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Q-Learning with Scalar Adjoint Matching
  2. 2026QAMM: Adjoint MeanFlow Matching for Few-Step Offline Reinforcement Learning
  3. 2026Discrete Adjoint Matching
  4. 2026Efficient Adjoint Matching for Fine-tuning Diffusion Models
  5. 2026Aligning Flow Map Policies with Optimal Q-Guidance
  6. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RL
  7. 2026Causal Flow Q-Learning for Robust Offline Reinforcement Learning
  8. 2025One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learning