Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork
Not on the map. Why?
Abstract (arXiv)
In-Context Reinforcement Learning (ICRL) has enabled foundation agents to adapt instantaneously to novel tasks, yet its efficacy in Ad-Hoc Teamwork (AHT)-where coordination with unknown partners is required-remains unexplored. To rigorously evaluate this, we introduce a large-scale benchmark ICRL4AHT, built upon a high-throughput JAX implementation of Overcooked-V2. Our benchmark includes a large, diverse teammate suite spanning both RL and heuristic policies, enabling controlled train-test shifts, and provides a reproducible end-to-end pipeline for teammate generation, learning-history collection, dataset construction, and online multi-episode evaluation. We evaluate representative history-conditioned ICRL algorithms, including Algorithm Distillation (AD) and Decision-Pretrained Transformer (DPT), across millions of transitions. Results reveal notable limitations: contrary to their success in single-agent domains, these baselines fail to exhibit robust test-time adaptation in multi-agent settings. Specifically, these methods frequently underperform random baselines across both unseen teammate and unseen layout tracks, with no clear in-context improvement over long horizons. These findings highlight the challenges of strategic inference under partial observability within the OvercookedV2 AHT protocol, establishing our benchmark as a critical testbed for next-generation coordination algorithms.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 54 of its 71 references that are in the map. First posted on arXiv 23 May 2026.
Builds on
54 of 71 references.
References · 54
- 2017Attention Is All You Needcited by 59,932
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2016Gaussian Error Linear Units (GELUs)cited by 3,431
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
- 2020On Layer Normalization in the Transformer Architecturecited by 906
- 2022A Generalist Agentcited by 802
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
- 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
- 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
- 2016Learning to reinforcement learncited by 369
- 2017Population Based Training of Neural Networkscited by 351
- 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
38 more
- 2021Transformers Can Do Bayesian Inferencecited by 338
- 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
- 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
- 2022Multi-Game Decision Transformerscited by 212
- 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 207
- 2023Large Language Models as General Pattern Machinescited by 196
- 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 194
- 2020"Other-Play" for Zero-Shot Coordinationcited by 194
- 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
- 2021Collaborating with Humans without Human Datacited by 139
- 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
- 2020Effective Diversity in Population Based Reinforcement Learningcited by 116
- 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
- 2023A Tutorial on Meta-Reinforcement Learningcited by 102
- 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
- 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 73
- 2021Off-Belief Learningcited by 72
- 2021Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordinationcited by 69
- 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environmentscited by 62
- 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 59
- 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
- 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
- 2022Melting Pot 2.0cited by 37
- 2023Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biasedcited by 37
- 2021Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomizationcited by 35
- 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
- 2021A New Formalism, Method and Open Issues for Zero-Shot Coordinationcited by 31
- 2022The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Modelscited by 31
- 2023Cooperative Open-ended Learning Framework for Zero-shot Coordinationcited by 30
- 2023Emergence of In-Context Reinforcement Learning from Noise Distillationcited by 26
- 2021Evaluating the Robustness of Collaborative Agentscited by 25
- 2024Learning to Cooperate with Humans using Generative Agentscited by 23
- 2022Addressing Optimism Bias in Sequence Modeling for Reinforcement Learningcited by 22
- 2022Generating Teammates for Training Robust Ad Hoc Teamwork Agents via Best-Response Diversitycited by 17
- 2023Diverse Conventions for Human-AI Collaborationcited by 14
- 2025OvercookedV2: Rethinking Overcooked for Zero-Shot Coordinationcited by 14
- 2023Minimum Coverage Sets for Training Robust Ad Hoc Teamwork Agentscited by 13
- 2024XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learningcited by 13
50 most cited of 54 All 54 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.