mlsanityData to 14 Oct 2026

In-Context Learning Enables Robot Action Prediction in LLMs

arXiv 2410.12782 ↗First posted 16 Oct 2024cs.RO cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

Recently, Large Language Models (LLMs) have achieved remarkable success using in-context learning (ICL) in the language domain. However, leveraging the ICL capabilities within LLMs to directly predict robot actions remains largely unexplored. In this paper, we introduce RoboPrompt, a framework that enables off-the-shelf text-only LLMs to directly predict robot actions through ICL without training. Our approach first heuristically identifies keyframes that capture important moments from an episode. Next, we extract end-effector actions from these keyframes as well as the estimated initial object poses, and both are converted into textual descriptions. Finally, we construct a structured template to form ICL demonstrations from these textual descriptions and a task instruction. This enables an LLM to directly predict robot actions at test time. Through extensive experiments and analysis, RoboPrompt shows stronger performance over zero-shot and ICL baselines in simulated and real-world settings. Our project page is available at https://davidyyd.github.io/roboprompt.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 60 of its 66 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

60 of 66 references.

References · 60

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Segment Anythingcited by 8,234
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
38 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  4. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  7. 2024Qwen2 Technical Reportcited by 2,120
  8. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  9. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  13. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  14. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  15. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  16. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  17. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  18. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  19. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  20. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  21. 2023FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objectscited by 429
  22. 2022Visual Language Maps for Robot Navigationcited by 365
  23. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  24. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
  25. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  26. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  27. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  28. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  29. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  30. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 224
  31. 2023Large Language Models as General Pattern Machinescited by 197
  32. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  33. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  34. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 148
  35. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 139
  36. 2023Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulationcited by 138
  37. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 137
  38. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 112

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 12.

27 papers.

Sort:

By area:

  1. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 221
  2. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 156
  3. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 33
  4. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  5. 2025Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steeringcited by 29
  6. 2025MAP-VLA: Memory-Augmented Prompting for Vision-Language-Action Model in Robotic Manipulationcited by 28
  7. 2025RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Modelscited by 22
  8. 2026Steerable Vision-Language-Action Policies for Embodied Reasoning and Hierarchical Controlcited by 20
  9. 2025Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrationscited by 15
  10. 2026An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyondcited by 11
  11. 2025VLMgineer: Vision Language Models as Robotic Toolsmithscited by 10
  12. 2025Maestro: Orchestrating Robotics Modules with Vision-Language Models for Zero-Shot Generalist Robotscited by 9
15 more
  1. 2026In-Context Robot Learning with VLM Agentscited by 8
  2. 2025Robot Operation of Home Appliances by Reading User Manualscited by 4
  3. 2026SAIL: Test-Time Scaling for In-Context Imitation Learning with VLMcited by 3
  4. 2025MaP-AVR: A Meta-Action Planner for Agents Leveraging Vision Language Models and Retrieval-Augmented Generationcited by 1
  5. 2026A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Roboticscited by 1
  6. 2026Bimanual Robot Manipulation via Multi-Agent In-Context Learningcited by 1
  7. 2026ContextFlow: In-Context Flow Matching for Robot Manipulationcited by 1
  8. 2026In-Context Learning for Robots: Methods and Applicationscited by 1
  9. 2025Boosting Embodied AI Agents through Perception-Generation Disaggregation and Asynchronous Pipeline Executioncited by 0
  10. 2025LEO-RobotAgent: A General-purpose Robotic Agent for Language-driven Embodied Operatorcited by 0
  11. 2025Few-Shot Inference of Human Perceptions of Robot Performance in Social Navigation Scenarioscited by 0
  12. 2026Decompose and Recompose: Reasoning New Skills from Existing Abilities for Cross-Task Robotic Manipulationcited by 0
  13. 2026CLASP: Language-Driven Robot Skill Selection and Composition using Task-Parameterized Learningcited by 0
  14. 2026RoboICL: Embodied In-Context Learning with GPT-6 Astracited by 0
  15. 2026PopGen: Generating Semantically Diverse Populations for Zero-Shot Coordinationcited by 0