mlsanityData to 14 Oct 2026

Yes, Q-learning Helps Offline In-Context RL

arXiv 2502.17666 ↗First posted 24 Feb 2025cs.LG cs.AIcited by 10

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Existing offline in-context reinforcement learning (ICRL) methods have predominantly relied on supervised training objectives, which are known to have limitations in offline RL settings. In this study, we explore the integration of RL objectives within an offline ICRL framework. Through experiments on more than 150 GridWorld and MuJoCo environment-derived datasets, we demonstrate that optimizing RL objectives directly improves performance by approximately 30% on average compared to widely adopted Algorithm Distillation (AD), across various dataset coverages, structures, expertise levels, and environmental complexities. Furthermore, in the challenging XLand-MiniGrid environment, RL objectives doubled the performance of AD. Our results also reveal that the addition of conservatism during value learning brings additional improvements in almost all settings tested. Our findings emphasize the importance of aligning ICRL learning objectives with the RL reward-maximization goal, and demonstrate that offline RL is a promising direction for advancing ICRL.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 48 of its 54 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

48 of 54 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?10 Jun
  2. 2024In-context Exploration-Exploitation for Reinforcement Learning11 Mar
  3. 2023Katakomba: Tools and Benchmarks for Data-Driven NetHack14 Jun
  4. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  5. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  6. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?12 Apr
  7. 2021A Dataset Perspective on Offline Reinforcement Learning8 Nov
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  10. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  11. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
1 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2016Layer Normalizationcited by 5,669
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  12. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
23 more
  1. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  2. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  3. 2020The NetHack Learning Environmentcited by 198
  4. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  5. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  6. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  7. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  8. 2023A Tutorial on Meta-Reinforcement Learningcited by 108
  9. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  10. 2024In-Context Language Learning: Architectures and Algorithmscited by 89
  11. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 78
  12. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 55
  13. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  14. 2025A Survey of In-Context Reinforcement Learningcited by 39
  15. 2024Q-value Regularized Transformer for Offline Reinforcement Learningcited by 29
  16. 2023Emergence of In-Context Reinforcement Learning from Noise Distillationcited by 27
  17. 2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thoughtcited by 26
  18. 2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
  19. 2023In-Context Reinforcement Learning for Variable Action Spacescited by 24
  20. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 21
  21. 2024XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learningcited by 14
  22. 2025Vintix: Action Model via In-Context Reinforcement Learningcited by 10
  23. 2024N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamicscited by 8
  2. 2025Mixture-of-Experts Meets In-Context Reinforcement Learningcited by 8
  3. 2025Scalable In-Context Q-Learningcited by 5
  4. 2025Towards Provable Emergence of In-Context Reinforcement Learningcited by 3
  5. 2025Dejavu: Towards Experience Feedback Learning for Embodied Intelligencecited by 3
  6. 2026Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamworkcited by 2
  7. 2026Vintix II: Decision Pre-Trained Transformer is a Scalable In-Context Reinforcement Learnercited by 1
  8. 2026In-Context Reinforcement Learning under Non-Stationarity: A Surveycited by 1
  9. 2025Context Value Informed In-Context Reinforcement Learningcited by 0
  10. 2026Learning in Context, Guided by Choice: A Reward-Free Paradigm for Reinforcement Learning with Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  2. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  3. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  4. 2025NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios
  5. 2025In-Context Compositional Q-Learning for Offline Reinforcement Learning
  6. 2026Active Offline-to-Online Reinforcement Learning
  7. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  8. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems