mlsanityData to 7 Oct 2026

Return-to-Go Is More Than a Number: Q-Guided Alignment for Return-Conditioned Supervised Learning

arXiv 2605.29028 ↗First posted 27 May 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Conditioned Sequence Models (CSMs) learn policies by treating return-to-go (RTG) as a control signal. However, existing CSMs often treat the RTGs as simple numerical inputs rather than aligning them with the performance of their policies. In this paper, we propose Q-ALIGN DT, a framework that enforces this alignment by ensuring the Q-value of the output policy is consistent with the input RTG. By leveraging a Q function to provide dense guidance to CSMs and further fine-tuning it using an RTG-perturbation technique with the CSM, our method ensures that higher RTGs are consistently mapped to trajectories with higher expected returns. Theoretically, we show that Q-ALIGN DT can efficiently learn the desired policy and output a near-optimal one when the RTG is sufficiently high. Empirically, we demonstrate through extensive experiments that Q-ALIGN DT achieves superior controllability and performance across the D4RL benchmark. Remarkably, our model effectively learns a structured family of policies that maintains precise alignment and generalizes to tasks like velocity-tracking where prior methods fail.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 51 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

43 of 51 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  10. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  11. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
31 more
  1. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  6. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  7. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  8. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  9. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  10. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  11. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  12. 2020Critic Regularized Regressioncited by 269
  13. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  14. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  15. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  16. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  17. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  18. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  19. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  20. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  21. 2023A Tutorial on Meta-Reinforcement Learningcited by 102
  22. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
  23. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
  24. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 73
  25. 2023Elastic Decision Transformercited by 66
  26. 2024Q-value Regularized Transformer for Offline Reinforcement Learningcited by 28
  27. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Makingcited by 26
  28. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learningcited by 25
  29. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  30. 2024Adaptive Q-Aid for Conditional Supervised Learning in Offline Reinforcement Learningcited by 7
  31. 2024Return-Aligned Decision Transformercited by 3

Built on by

Not cited yet.