mlsanityData to 7 Oct 2026

In-Context Imitation Learning via Next-Token Prediction

arXiv 2408.15980 ↗First posted 28 Aug 2024cs.RO cs.AIcited by 65

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

We explore how to enhance next-token prediction models to perform in-context imitation learning on a real robot, where the robot executes new tasks by interpreting contextual information provided during the input phase, without updating its underlying policy parameters. We propose In-Context Robot Transformer (ICRT), a causal transformer that performs autoregressive prediction on sensorimotor trajectories without relying on any linguistic data or reward function. This formulation enables flexible and training-free execution of new tasks at test time, achieved by prompting the model with sensorimotor trajectories of the new task composing of image observations, actions and states tuples, collected through human teleoperation. Experiments with a Franka Emika robot demonstrate that the ICRT can adapt to new tasks specified by prompts, even in environment configurations that differ from both the prompt and the training data. In a multitask environment setup, ICRT significantly outperforms current state-of-the-art next-token prediction models in robotics on generalizing to unseen tasks. Code, checkpoints and data are available on https://icrt.dev/

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 56 of its 68 references that are in the map. First posted on arXiv 28 Aug 2024.

Builds on

56 of 68 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics28 Mar
  2. 2023BridgeData V2: A Dataset for Robot Learning at Scale24 Aug
  3. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware23 Apr
  4. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training30 Sep
  5. 2022Demonstrate Once, Imitate Immediately (DOME): Learning Visual Servoing for One-Shot Imitation Learning6 Apr
  6. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb
  7. 2021Towards More Generalizable One-shot Visual Imitation Learning26 Oct

Other references · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  12. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
37 more
  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  5. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  6. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  7. 2022A Generalist Agentcited by 802
  8. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  10. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  11. 2021Implicit Behavioral Cloningcited by 442
  12. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  13. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  14. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  15. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  16. 2022Masked Visual Pre-training for Motor Controlcited by 246
  17. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  18. 2017One-Shot Imitation Learningcited by 232
  19. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  20. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  21. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  22. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  23. 2023ViNT: A Foundation Model for Visual Navigationcited by 210
  24. 2023Large Language Models as General Pattern Machinescited by 196
  25. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  26. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  27. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  28. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  29. 2020ACRONYM: A Large-Scale Grasp Dataset Based on Simulationcited by 115
  30. 2018Jacquard: A Large Scale Dataset for Robotic Grasp Detectioncited by 86
  31. 2024A Touch, Vision, and Language Dataset for Multimodal Alignmentcited by 68
  32. 2024Humanoid Locomotion as Next Token Predictioncited by 63
  33. 2023Robot Learning with Sensorimotor Pre-trainingcited by 51
  34. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 47
  35. 2023Hyper-Decision Transformer for Efficient Online Policy Adaptationcited by 32
  36. 2024Rethinking Patch Dependence for Masked Autoencoderscited by 29
  37. 2024SUGAR: Pre-training 3D Visual Representations for Roboticscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 31; 2026: 28.

65 papers.

Sort:

By area:

  1. 2024Vision Language Models are In-Context Value Learnerscited by 84
  2. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  3. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulationcited by 72
  4. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  5. 2025OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extractioncited by 54
  6. 2025AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real Worldcited by 46
  7. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  8. 2024Instant Policy: In-Context Imitation Learning via Graph Diffusioncited by 38
  9. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  10. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 31
  11. 2025Learning Long-Context Diffusion Policies via Past-Token Predictioncited by 28
  12. 2025MAP-VLA: Memory-Augmented Prompting for Vision-Language-Action Model in Robotic Manipulationcited by 28
38 more
  1. 2025VITA: Vision-to-Action Flow Matching Policycited by 23
  2. 2025Dense Policy: Bidirectional Autoregressive Learning of Actionscited by 19
  3. 2025RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Modelscited by 19
  4. 2025Robust Finetuning of Vision-Language-Action Robot Policies via Parameter Mergingcited by 19
  5. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Controlcited by 19
  6. 2025RobotArena ∞: Scalable Robot Benchmarking via Real-to-Sim Translationcited by 15
  7. 2024Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Resetcited by 14
  8. 2026Gated Memory Policy: In-Context Memorization and Adaptationcited by 13
  9. 2026RoboTTT: Context Scaling for Robot Policiescited by 13
  10. 2025Action Tokenizer Matters in In-Context Imitation Learningcited by 12
  11. 2025Physical Autoregressive Model for Robotic Manipulation without Action Pretrainingcited by 11
  12. 2024Dynamics as Prompts: In-Context Learning for Sim-to-Real System Identificationscited by 10
  13. 2025MTIL: Encoding Full History with Mamba for Temporal Imitation Learningcited by 10
  14. 2025FreqPolicy: Frequency Autoregressive Visuomotor Policy with Continuous Tokenscited by 9
  15. 2025CDP: Towards Robust Autoregressive Visuomotor Policy Learning via Causal Diffusioncited by 8
  16. 2025cVLA: Towards Efficient Camera-Space VLAscited by 8
  17. 2025Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Compositioncited by 8
  18. 2026An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyondcited by 8
  19. 2025Learning Affordances at Inference-Time for Vision-Language-Action Modelscited by 6
  20. 2026VPWEM: Non-Markovian Visuomotor Policy with Working and Episodic Memorycited by 6
  21. 2026Breaking Lock-In: Preserving Steerability under Low-Data VLA Post-Trainingcited by 6
  22. 2025RoboSSM: Scalable In-context Imitation Learning via State-Space Modelscited by 5
  23. 2026From Abstraction to Instantiation: Learning Behavioral Representation for Vision-Language-Action Modelcited by 5
  24. 2025Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worldscited by 4
  25. 2025Behavioral Exploration: Learning to Explore via In-Context Adaptationcited by 4
  26. 2025MoS-VLA: A Vision-Language-Action Model with One-Shot Skill Adaptationcited by 4
  27. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  28. 2025The Temporal Trap: Entanglement in Pre-Trained Visual Representations for Visuomotor Policy Learningcited by 3
  29. 2026Distinct mechanisms underlying in-context learning in transformerscited by 3
  30. 2025Robo-DM: Data Management For Large Robot Datasetscited by 2
  31. 2025Robust Instant Policy: Leveraging Student's t-Regression Model for Robust In-context Imitation Learning of Robot Manipulationcited by 2
  32. 2025Attentive Feature Aggregation or: How Policies Learn to Stop Worrying about Robustness and Attend to Task-Relevant Visual Cuescited by 2
  33. 2026SAIL: Test-Time Scaling for In-Context Imitation Learning with VLMcited by 2
  34. 2026In-Context World Modeling for Robotic Controlcited by 2
  35. 2026Ordered Action Tokens for Visuomotor Policy Learningcited by 2
  36. 2026Instant-Fold: In-Context Imitation Learning for Deformable Object Manipulationcited by 1
  37. 2026Decoupling the Declarative from the Procedural in Vision-Language-Action Modelscited by 1
  38. 2026StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Modelscited by 1

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ICLR: In-Context Imitation Learning with Visual Reasoning
  2. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learning
  3. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data
  4. 2025Task Tokens: A Flexible Approach to Adapting Behavior Foundation Models
  5. 2026ConceptACT: Episode-Level Concepts for Sample-Efficient Robotic Imitation Learning
  6. 2025CIVIL: Causal and Intuitive Visual Imitation Learning
  7. 2026Training and Evaluating Diffusion Policies with Long Context Lengths
  8. 2025In-Context Iterative Policy Improvement for Dynamic Manipulation