mlsanityData to 7 Oct 2026

Trust Region Q Adjoint Matching

arXiv 2605.27079 ↗First posted 26 May 2026cs.LG cs.AI cs.ROcited by 3

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Off-policy reinforcement learning of pretrained flow policies remains challenging due to the instability of optimization arising from the multi-step sampling process. Recently, Q-learning with Adjoint Matching (QAM) addressed this by recasting policy improvement as a stochastic optimal control (SOC) problem guided by a learned critic. However, QAM inherits a fundamental fragility of critic-guided improvement, since small critic errors can be exponentially amplified and often lead to performance collapse. This paper introduces Trust Region Q Adjoint Matching (TRQAM), a stable off-policy fine-tuning algorithm that adaptively controls the path-space KL between the fine-tuned and pretrained policies through projected dual descent. Specifically, we adapt a trust-region parameter λ inside the sampling process of the flow policy and prove that λ exactly weights the path-space KL in the SOC objective. As a result, our method can tightly control the deviation from the pretrained policy within a prescribed KL budget, achieving stable off-policy RL. Through experiments on 50 OGBench tasks, TRQAM consistently outperforms prior methods in both offline RL and offline-to-online RL. In particular, TRQAM achieves an overall success rate of 68% in offline RL, substantially improving over the strongest baseline at 46%.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 49 of its 56 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

49 of 56 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations24 Sep
  2. 2025Flow Q-Learning4 Feb
  3. 2024OGBench: Benchmarking Offline Goal-Conditioned RL26 Oct
  4. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  5. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 44

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  3. 2022Flow Matching for Generative Modelingcited by 5,426
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  9. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  10. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  11. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  12. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
32 more
  1. 2023Stochastic Interpolants: A Unifying Framework for Flows and Diffusionscited by 897
  2. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  3. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 707
  4. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  8. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  9. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  10. 2018Maximum a Posteriori Policy Optimisationcited by 318
  11. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  12. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285
  13. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  14. 2024Diffusion Policy Policy Optimizationcited by 255
  15. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  16. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 222
  17. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  18. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  19. 2025Reinforcement Learning with Action Chunkingcited by 97
  20. 2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learningcited by 83
  21. 2020Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path spacecited by 80
  22. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 72
  23. 2023Imitation Bootstrapped Reinforcement Learningcited by 54
  24. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Modelscited by 43
  25. 2023Stochastic Optimal Control Matchingcited by 39
  26. 2026Q-learning with Adjoint Matchingcited by 33
  27. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  28. 2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 27
  29. 2025NORA-1.5: A Vision-Language-Action Model Trained using World Model- and Action-based Preference Rewardscited by 26
  30. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
  31. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  32. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policiescited by 2
  2. 2026QAMM: Adjoint MeanFlow Matching for Few-Step Offline Reinforcement Learningcited by 0
  3. 2026Diffusion Policy Improvement with Proposal-Conditioned Refinement Flowscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Q-Learning with Scalar Adjoint Matching
  2. 2026Aligning Flow Map Policies with Optimal Q-Guidance
  3. 2026Entropy-Regularized Adjoint Matching for Offline Reinforcement Learning
  4. 2026Flow Matching for Offline Reinforcement Learning with Discrete Actions
  5. 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning
  6. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
  7. 2023Supported Trust Region Optimization for Offline Reinforcement Learning
  8. 2026GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policies