mlsanityData to 7 Oct 2026

Adaptive Q-Aid for Conditional Supervised Learning in Offline Reinforcement Learning

arXiv 2402.02017 ↗First posted 3 Feb 2024cs.LGcited by 7

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) has progressed with return-conditioned supervised learning (RCSL), but its lack of stitching ability remains a limitation. We introduce Q-Aided Conditional Supervised Learning (QCS), which effectively combines the stability of RCSL with the stitching capability of Q-functions. By analyzing Q-function over-generalization, which impairs stable stitching, QCS adaptively integrates Q-aid into RCSL's loss function based on trajectory return. Empirical results show that QCS significantly outperforms RCSL and value-based methods, consistently achieving or exceeding the maximum trajectory returns across diverse offline RL benchmarks.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 56 references that are in the map. First posted on arXiv 3 Feb 2024.

Builds on

46 of 56 references.

Closest ancestors · 19

Same direction, newest first

  1. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning27 Oct
  2. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL6 Oct
  3. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  4. 2022When does return-conditioned supervised learning work for offline reinforcement learning?2 Jun
  5. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?12 Apr
  6. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  7. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  8. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization9 Dec
  9. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  10. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  11. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  12. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
7 more
  1. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  2. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  3. 2020Critic Regularized Regression26 Jun
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2016Layer Normalizationcited by 5,579
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  12. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
15 more
  1. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  2. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  4. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  5. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  6. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  7. 2022Online Decision Transformercited by 194
  8. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  9. 2023Elastic Decision Transformercited by 66
  10. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  11. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  12. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Makingcited by 26
  13. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learningcited by 25
  14. 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
  15. 2023Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 38
  2. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
  3. 2024OffRIPP: Offline RL-based Informative Path Planningcited by 1
  4. 2025B3C: A Minimalist Approach to Offline Multi-Agent Reinforcement Learningcited by 1
  5. 2026Proximal Action Replacement for Behavior Cloning Actor-Critic in Offline Reinforcement Learningcited by 1
  6. 2026Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learningcited by 0
  7. 2026Return-to-Go Is More Than a Number: Q-Guided Alignment for Return-Conditioned Supervised Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Closing the Gap between TD Learning and Supervised Learning with Q-Conditioned Maximization
  2. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  3. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  4. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  5. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  6. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning
  7. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning
  8. 2026Q-learning Penalized Transformer for Safe Offline Reinforcement Learning