mlsanityData to 14 Oct 2026

Scalable In-Context Q-Learning

arXiv 2506.01299 ↗First posted 2 Jun 2025cs.AI cs.LGcited by 5

On the map: World models → Latent dynamics models and model-based RL

also in 1 more
Abstract (arXiv)

Recent advancements in language models have demonstrated remarkable in-context learning abilities, prompting the exploration of in-context reinforcement learning (ICRL) to extend the promise to decision domains. Due to involving more complex dynamics and temporal correlations, existing ICRL approaches may face challenges in learning from suboptimal trajectories and achieving precise in-context inference. In the paper, we propose Scalable In-Context Q-Learning (S-ICQL), an innovative framework that harnesses dynamic programming and world modeling to steer ICRL toward efficient reward maximization and task generalization, while retaining the scalability and stability of supervised pretraining. We design a prompt-based multi-head transformer architecture that simultaneously predicts optimal policies and in-context value functions using separate heads. We pretrain a generalized world model to capture task-relevant information, enabling the construction of a compact prompt that facilitates fast and precise in-context inference. During training, we perform iterative policy improvement by fitting a state value function to an upper-expectile of the Q-function, and distill the in-context value functions into policy extraction using advantage-weighted regression. Extensive experiments across a range of discrete and continuous environments show consistent performance gains over various types of baselines, especially when learning from suboptimal data. Our code is available at textcolormagentahrefhttps://github.com/NJU-RL/SICQLhttps://github.com/NJU-RL/SICQL.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 45 of its 59 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

45 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglement15 Oct
  2. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning8 Aug

Other references · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  9. 2018Exploration by Random Network Distillationcited by 993
  10. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  12. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
31 more
  1. 2025Learning to Reason under Off-Policy Guidancecited by 281
  2. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  3. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  4. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  5. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  6. 2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 118
  7. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 111
  8. 2023A Tutorial on Meta-Reinforcement Learningcited by 108
  9. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  10. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 98
  11. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 93
  12. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  13. 2023MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RLcited by 68
  14. 2023Visual In-Context Promptingcited by 52
  15. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 41
  16. 2025A Survey of In-Context Reinforcement Learningcited by 39
  17. 2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learningcited by 35
  18. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  19. 2023Emergence of In-Context Reinforcement Learning from Noise Distillationcited by 27
  20. 2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thoughtcited by 26
  21. 2023In-Context Reinforcement Learning for Variable Action Spacescited by 24
  22. 2020Lifelong Incremental Reinforcement Learning with Online Bayesian Inferencecited by 23
  23. 2023Context Shift Reduction for Offline Meta-Reinforcement Learningcited by 18
  24. 2024Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learningcited by 17
  25. 2024In-context Exploration-Exploitation for Reinforcement Learningcited by 16
  26. 2022A Dirichlet Process Mixture of Robust Task Models for Scalable Lifelong Reinforcement Learningcited by 15
  27. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 15
  28. 2024XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learningcited by 14
  29. 2025Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planningcited by 13
  30. 2025Yes, Q-learning Helps Offline In-Context RLcited by 10
  31. 2025Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervisioncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Mixture-of-Experts Meets In-Context Reinforcement Learningcited by 8
  2. 2026DADP: Domain Adaptive Diffusion Policycited by 1
  3. 2026Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learningcited by 1
  4. 2025Context Value Informed In-Context Reinforcement Learningcited by 0
  5. 2026Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026In-Context Reinforcement Learning under Non-Stationarity: A Survey
  2. 2025Vintix: Action Model via In-Context Reinforcement Learning
  3. 2025Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds
  4. 2026TR-ICRL: Test-Time Rethinking for In-Context Reinforcement Learning
  5. 2025In-Context Compositional Q-Learning for Offline Reinforcement Learning
  6. 2022Large Language Models can Implement Policy Iteration
  7. 2026Learning in Context, Guided by Choice: A Reward-Free Paradigm for Reinforcement Learning with Transformers
  8. 2026Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement