mlsanityData to 14 Oct 2026

Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics

arXiv 2403.19578 ↗First posted 28 Mar 2024cs.RO cs.LG cs.NEcited by 74

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

We show that off-the-shelf text-based Transformers, with no additional training, can perform few-shot in-context visual imitation learning, mapping visual observations to action sequences that emulate the demonstrator's behaviour. We achieve this by transforming visual observations (inputs) and trajectories of actions (outputs) into sequences of tokens that a text-pretrained Transformer (GPT-4 Turbo) can ingest and generate, via a framework we call Keypoint Action Tokens (KAT). Despite being trained only on language, we show that these Transformers excel at translating tokenised visual keypoint observations into action trajectories, performing on par or better than state-of-the-art imitation learning (diffusion policies) in the low-data regime on a suite of real-world, everyday tasks. Rather than operating in the language domain as is typical, KAT leverages text-based Transformers to operate in the vision and action domains to learn general patterns in demonstration data for highly efficient imitation learning, indicating promising new avenues for repurposing natural language models for embodied tasks. Videos are available at https://www.robot-learning.uk/keypoint-action-tokens.

Key facts. Cited by 74 papers in the map (data through 14 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 40 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

27 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Few-Shot In-Context Imitation Learning via Implicit Graph Alignment18 Oct
  2. 2018Task-Embedded Control Networks for Few-Shot Imitation Learning8 Oct

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  10. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  12. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
13 more
  1. 2022In-context Learning and Induction Headscited by 964
  2. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  3. 2022Transformers learn in-context by gradient descentcited by 732
  4. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  5. 2021Deep ViT Features as Dense Visual Descriptorscited by 330
  6. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  7. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
  8. 2017One-Shot Imitation Learningcited by 233
  9. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 101
  10. 2023Language Models as Zero-Shot Trajectory Generatorscited by 60
  11. 2023Train Offline, Test Online: A Real Robot Learning Benchmarkcited by 47
  12. 2023An Unbiased Look at Datasets for Visuo-Motor Pre-Trainingcited by 42
  13. 2020Action Image Representation: Learning Scalable Deep Grasping Policies with Zero Real World Datacited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 29; 2026: 28.

74 papers.

Sort:

By area:

  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
  2. 2024Vision Language Models are In-Context Value Learnerscited by 90
  3. 2024In-Context Imitation Learning via Next-Token Predictioncited by 70
  4. 2025VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizerscited by 70
  5. 2024GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMscited by 52
  6. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 50
  7. 2024FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Modelcited by 42
  8. 2024Instant Policy: In-Context Imitation Learning via Graph Diffusioncited by 39
  9. 2024R+X: Retrieval and Execution from Everyday Human Videoscited by 38
  10. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 33
  11. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  12. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 27
38 more
  1. 2025ReconDreamer-RL: Enhancing Reinforcement Learning via Diffusion-based Scene Reconstructioncited by 27
  2. 2025Robotic Visual Instructioncited by 22
  3. 2025RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Modelscited by 22
  4. 2025SKIL: Semantic Keypoint Imitation Learning for Generalizable Data-efficient Manipulationcited by 21
  5. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 18
  6. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  7. 2024STEER: Flexible Robotic Manipulation via Dense Language Groundingcited by 14
  8. 2025Action Tokenizer Matters in In-Context Imitation Learningcited by 14
  9. 2024Learning from 10 Demos: Generalisable and Sample-Efficient Policy Learning with Oriented Affordance Framescited by 10
  10. 2024Dynamics as Prompts: In-Context Learning for Sim-to-Real System Identificationscited by 10
  11. 2025VLMgineer: Vision Language Models as Robotic Toolsmithscited by 10
  12. 2026ABPolicy: Asynchronous B-Spline Flow Policy for Real-Time and Smooth Robotic Manipulationcited by 10
  13. 2025Maestro: Orchestrating Robotics Modules with Vision-Language Models for Zero-Shot Generalist Robotscited by 9
  14. 2025One-Shot Dual-Arm Imitation Learningcited by 8
  15. 2025CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguitycited by 8
  16. 2025cVLA: Towards Efficient Camera-Space VLAscited by 8
  17. 2024Generative AI Agents in Autonomous Machines: A Safety Perspectivecited by 7
  18. 2025Imitation Learning from a Single Temporally Misaligned Videocited by 7
  19. 2024EMOTION: Expressive Motion Sequence Generation for Humanoid Robots with In-Context Learningcited by 6
  20. 2024Select2Plan: Training-Free ICL-Based Planning through VQA and Memory Retrievalcited by 5
  21. 2025A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewardscited by 5
  22. 2025RoboSSM: Scalable In-context Imitation Learning via State-Space Modelscited by 5
  23. 2025Distracted Robot: How Visual Clutter Undermine Robotic Manipulationcited by 5
  24. 2026Robot Learning from Human Videos: A Surveycited by 5
  25. 2024TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Modelscited by 4
  26. 2026Scaling Single Human Demonstrations for Imitation Learning using Generative Foundational Modelscited by 4
  27. 2025Robust Instant Policy: Leveraging Student's t-Regression Model for Robust In-context Imitation Learning of Robot Manipulationcited by 3
  28. 2025MaNGO - Adaptable Graph Network Simulators via Meta-Learningcited by 3
  29. 2025Improving Robotic Manipulation Robustness via NICE Scene Surgerycited by 3
  30. 2026SAIL: Test-Time Scaling for In-Context Imitation Learning with VLMcited by 3
  31. 2026Ordered Action Tokens for Visuomotor Policy Learningcited by 3
  32. 2025Knowledge-Driven Imitation Learning: Enabling Generalization Across Diverse Conditionscited by 2
  33. 2025Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)cited by 2
  34. 2026Tether: Autonomous Functional Play with Correspondence-Driven Trajectory Warpingcited by 2
  35. 2026On the Generalization Capabilities, Design Choices and Limitations of Keypoint Imitation Learningcited by 2
  36. 2026Instant-Fold: In-Context Imitation Learning for Deformable Object Manipulationcited by 2
  37. 2026Improving Vision-Language-Action Model Fine-Tuning with Structured Stage and Keyframe Supervisioncited by 2
  38. 2025Reducing Latency in LLM-Based Natural Language Commands Processing for Robot Navigationcited by 1

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026WorldToken: Time-First Sequence Modeling for Robotic Imitation Learning
  2. 2026Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
  3. 2022K-VIL: Keypoints-based Visual Imitation Learning
  4. 2024VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions
  5. 2026Native Video-Action Pretraining for Generalizable Robot Control
  6. 2025LaVA-Man: Learning Visual Action Representations for Robot Manipulation
  7. 2026ICLR: In-Context Imitation Learning with Visual Reasoning
  8. 2025Learning Compositional Behaviors from Demonstration and Language