mlsanityData to 7 Oct 2026

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations

arXiv 2412.01441 ↗First posted 2 Dec 2024cs.AI cs.LGcited by 32

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

In this paper, we present a benchmark to pressure-test today's frontier models' multimodal decision-making capabilities in the very long-context regime (up to one million tokens) and investigate whether these models can learn from large numbers of expert demonstrations in their context. We evaluate the performance of Claude 3.5 Sonnet, Gemini 1.5 Flash, Gemini 1.5 Pro, Gemini 2.0 Flash Experimental, GPT-4o, o1-mini, o1-preview, and o1 as policies across a battery of simple interactive decision-making tasks: playing tic-tac-toe, chess, and Atari, navigating grid worlds, solving crosswords, and controlling a simulated cheetah. We study increasing amounts of expert demonstrations in the context unicodex2013 from no demonstrations to 512 full episodes. Across our tasks, models rarely manage to fully reach expert performance, and often, presenting more demonstrations has little effect. Some models steadily improve with more demonstrations on a few tasks. We investigate the effect of encoding observations as text or images and the impact of chain-of-thought prompting. To help quantify the impact of other approaches and future innovations, we open source our benchmark that covers the zero-, few-, and many-shot regimes in a unified evaluation.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 49 of its 72 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

49 of 72 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024In-Context Imitation Learning via Next-Token Prediction28 Aug
  2. 2024Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics28 Mar

Other references · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  9. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  11. 2018DeepMind Control Suitecited by 935
  12. 2023Large Language Models as Optimizerscited by 830
35 more
  1. 2022A Generalist Agentcited by 802
  2. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  3. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  4. 2017One-Shot Imitation Learningcited by 232
  5. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 227
  6. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  7. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  8. 2024Many-Shot In-Context Learningcited by 203
  9. 2022Recurrent Memory Transformercited by 197
  10. 2020The NetHack Learning Environmentcited by 196
  11. 2023Large Language Models as General Pattern Machinescited by 196
  12. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  13. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  14. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  15. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 141
  16. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 138
  17. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  18. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  19. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  20. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agentscited by 89
  21. 2024Vision Language Models are In-Context Value Learnerscited by 84
  22. 2019Meta-learning of Sequential Strategiescited by 68
  23. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  24. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 64
  25. 2024Amortized Planning with Large-Scale Transformers: A Case Study on Chesscited by 45
  26. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  27. 2023Prompt a Robot to Walk with Large Language Modelscited by 44
  28. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  29. 2020Meta-trained agents implement Bayes-optimal agentscited by 37
  30. 2022Large Language Models can Implement Policy Iterationcited by 31
  31. 2024Learning Universal Predictorscited by 29
  32. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 25
  33. 2024Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agentcited by 20
  34. 2023Recurrent Action Transformer with Memorycited by 17
  35. 2023Memory-Based Meta-Learning on Non-Stationary Distributionscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 22; 2026: 8.

32 papers.

Sort:

By area:

  1. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 298
  2. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  3. 2025Agentic Large Language Models, a surveycited by 105
  4. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  5. 2026AutoHarness: improving LLM agents by automatically synthesizing a code harnesscited by 63
  6. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  7. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 36
  8. 2025VideoGameBench: Can Vision-Language Models complete popular video games?cited by 29
  9. 2025SIMA 2: A Generalist Embodied Agent for Virtual Worldscited by 26
  10. 2025PoE-World: Compositional World Modeling with Products of Programmatic Expertscited by 25
  11. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24
  12. 2025Synthesizing world models for bilevel planningcited by 13
20 more
  1. 2025TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Modelscited by 12
  2. 2025General Modular Harness for LLM Agents in Multi-Turn Gaming Environmentscited by 9
  3. 2026VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agentscited by 9
  4. 2024Can foundation models actively gather information in interactive environments to test hypotheses?cited by 7
  5. 2025MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning?cited by 7
  6. 2025FlySearch: Exploring how vision-language models explorecited by 7
  7. 2026The Illusion of Stochasticity in LLMscited by 7
  8. 2025Panoptic Captioning: An Equivalence Bridge for Image and Textcited by 6
  9. 2025Understanding Prompt Tuning and In-Context Learning via Meta-Learningcited by 6
  10. 2024Harnessing Language for Coordination: A Framework and Benchmark for LLM-Driven Multi-Agent Controlcited by 4
  11. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  12. 2025Can Large Language Models Master Complex Card Games?cited by 4
  13. 2025Beyond Needle(s) in the Embodied Haystack: Environment, Architecture, and Training Considerations for Long Context Reasoningcited by 3
  14. 2026Learning Abstractions for Hierarchical Planning in Program-Synthesis Agentscited by 3
  15. 2025Disentangling Exploration of Large Language Models by Optimal Exploitationcited by 1
  16. 2026Training Language Agents to Learn from Experiencecited by 1
  17. 2025Instruction Agent: Enhancing Agent with Expert Demonstrationcited by 0
  18. 2026PaperGuide: Making Small Language-Model Paper-Reading Agents More Efficientcited by 0
  19. 2026Emergent Misaligned Communication in Long-Horizon Multi-Agent LLM Commercecited by 0
  20. 2026V-ICAL Bench: Evaluating Video In-Context Learning for Multimodal Agents in Interactive Environmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RoboSSM: Scalable In-context Imitation Learning via State-Space Models
  2. 2022Eliciting Compatible Demonstrations for Multi-Human Imitation Learning
  3. 2026Training and Evaluating Diffusion Policies with Long Context Lengths
  4. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Play
  5. 2024From Introspection to Best Practices: Principled Analysis of Demonstrations in Multimodal In-Context Learning
  6. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitation
  7. 2026Difference-Aware Retrieval Policies for Imitation Learning
  8. 2024Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations