mlsanityData to 14 Oct 2026

In-context Exploration-Exploitation for Reinforcement Learning

arXiv 2403.06826 ↗First posted 11 Mar 2024cs.LG cs.AI stat.MLcited by 16

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

In-context learning is a promising approach for online policy learning of offline reinforcement learning (RL) methods, which can be achieved at inference time without gradient optimization. However, this method is hindered by significant computational costs resulting from the gathering of large training trajectory sets and the need to train large Transformer models. We address this challenge by introducing an In-context Exploration-Exploitation (ICEE) algorithm, designed to optimize the efficiency of in-context policy learning. Unlike existing models, ICEE performs an exploration-exploitation trade-off at inference time within a Transformer model, without the need for explicit Bayesian inference. Consequently, ICEE can solve Bayesian optimization problems as efficiently as Gaussian process biased methods do, but in significantly less time. Through experiments in grid world environments, we demonstrate that ICEE can learn to solve new RL tasks using only tens of episodes, marking a substantial improvement over the hundreds of episodes needed by the previous in-context learning method.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 52 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

33 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  2. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb

Other references · 31

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  6. 2022A Generalist Agentcited by 808
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  9. 2016Learning to learn by gradient descent by gradient descentcited by 620
  10. 2016Learning to reinforcement learncited by 377
  11. 2023Do Large Language Models Know What They Don't Know?cited by 243
  12. 2017One-Shot Imitation Learningcited by 233
19 more
  1. 2022Multi-Game Decision Transformerscited by 214
  2. 2022Online Decision Transformercited by 196
  3. 2016Learning to Optimizecited by 195
  4. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  5. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  6. 2018Meta-Gradient Reinforcement Learningcited by 159
  7. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  8. 2017Learned Optimizers that Scale and Generalizecited by 110
  9. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 102
  10. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 93
  11. 2018Unsupervised Learning via Meta-Learningcited by 82
  12. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 53
  13. 2020Meta-Gradient Reinforcement Learning with an Objective Discovered Onlinecited by 52
  14. 2019Discovery of Useful Questions as Auxiliary Taskscited by 45
  15. 2021Bootstrapped Meta-Learningcited by 45
  16. 2018Some Considerations on Learning to Explore via Meta-Reinforcement Learningcited by 44
  17. 2018Unsupervised Meta-Learning for Reinforcement Learningcited by 39
  18. 2023A Mechanism for Sample-Efficient In-Context Learning for Sparse Retrieval Taskscited by 17
  19. 2022A History of Meta-gradient: Gradient Methods for Meta-learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 3.

16 papers.

Sort:

By area:

  1. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 31
  2. 2025Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Taskscited by 19
  3. 2025Toward Efficient Exploration by Large Language Model Agentscited by 17
  4. 2025Yes, Q-learning Helps Offline In-Context RLcited by 10
  5. 2025Mixture-of-Experts Meets In-Context Reinforcement Learningcited by 8
  6. 2025Meta-Prompt Optimization for LLM-Based Sequential Decision Makingcited by 7
  7. 2025Large Language Model-Enhanced Multi-Armed Banditscited by 7
  8. 2025Scalable In-Context Q-Learningcited by 5
  9. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  10. 2025CoEx -- Co-evolving World-model and Explorationcited by 4
  11. 2025Towards Provable Emergence of In-Context Reinforcement Learningcited by 3
  12. 2025Safe In-Context Reinforcement Learningcited by 3
4 more
  1. 2026In-Context Reinforcement Learning through Bayesian Fusion of Context and Value Priorcited by 1
  2. 2026Learning to Learn from Language Feedback with Social Meta-Learningcited by 1
  3. 2026Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learningcited by 1
  4. 2025CooT: Learning to Coordinate In-Context with Coordination Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought
  2. 2026In-Context Reinforcement Learning under Non-Stationarity: A Survey
  3. 2024XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning
  4. 2025Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planning
  5. 2025Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds
  6. 2026In-Context Reinforcement Learning From Suboptimal Historical Data
  7. 2022Large Language Models can Implement Policy Iteration
  8. 2024A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learning