mlsanityData to 13 Oct 2026

Interactive Language: Talking to Robots in Real Time

arXiv 2210.06407 ↗First posted 12 Oct 2022cs.RO cs.AI cs.LGcited by 218

Not on the map. Why?

Cited by 37 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a framework for building interactive, real-time, natural language-instructable robots in the real world, and we open source related assets (dataset, environment, benchmark, and policies). Trained with behavioral cloning on a dataset of hundreds of thousands of language-annotated trajectories, a produced policy can proficiently execute an order of magnitude more commands than previous works: specifically we estimate a 93.5% success rate on a set of 87,000 unique natural language strings specifying raw end-to-end visuo-linguo-motor skills in the real world. We find that the same policy is capable of being guided by a human via real-time language to address a wide range of precise long-horizon rearrangement goals, e.g. "make a smiley face out of blocks". The dataset we release comprises nearly 600,000 language-labeled trajectories, an order of magnitude larger than prior available datasets. We hope the demonstrated results and associated assets enable further advancement of helpful, capable, natural-language-interactable robots. See videos at https://interactive-language.github.io.

Key facts. Cited by 218 papers in the map (data through 13 Oct 2026). Builds on 38 of its 63 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

38 of 63 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  6. 2017Deep reinforcement learning from human preferencescited by 3,871
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,321
  8. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,065
  9. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,032
  10. 2020On Layer Normalization in the Transformer Architecturecited by 910
  11. 2015End-to-End Training of Deep Visuomotor Policiescited by 850
  12. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 704
26 more
  1. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  2. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 571
  3. 2018On Evaluation of Embodied Navigation Agentscited by 531
  4. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 518
  5. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  6. 2021Implicit Behavioral Cloningcited by 442
  7. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  8. 2019Learning Latent Plans from Playcited by 313
  9. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  10. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  11. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 247
  12. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  13. 2019Goal-conditioned Imitation Learningcited by 160
  14. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  15. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  16. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  17. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 118
  18. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  19. 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
  20. 2022Correcting Robot Plans with Natural Language Feedbackcited by 68
  21. 2021Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learningcited by 40
  22. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  23. 2018Guiding Policies with Language via Meta-Learningcited by 34
  24. 2021LILA: Language-Informed Latent Actionscited by 27
  25. 2017Learning Interpretable Spatial Operations in a Rich 3D Blocks Worldcited by 15
  26. 2021Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Taskscited by 12

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 3; 2023: 51; 2024: 65; 2025: 63; 2026: 36.

218 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,022
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,823
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,052
  4. 2024Octo: An Open-Source Generalist Robot Policycited by 1,450
  5. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,433
  6. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,197
  7. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 739
  8. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 512
  9. 2023Learning Interactive Real-World Simulatorscited by 445
  10. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 418
  11. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 397
  12. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 357
38 more
  1. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 325
  2. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  3. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 313
  4. 2023Language to Rewards for Robotic Skill Synthesiscited by 309
  5. 2024Latent Action Pretraining from Videoscited by 308
  6. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  7. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 277
  8. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 265
  9. 2024RT-H: Action Hierarchies Using Languagecited by 207
  10. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 190
  11. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 181
  12. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 179
  13. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 171
  14. 2023Video Language Planningcited by 161
  15. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  16. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 146
  17. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 140
  18. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 132
  19. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 129
  20. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 125
  21. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 121
  22. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 118
  23. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  24. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  25. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 109
  26. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 106
  27. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 102
  28. 2023Consciousness in Artificial Intelligence: Insights from the Science of Consciousnesscited by 101
  29. 2023On Bringing Robots Homecited by 101
  30. 2024IRASim: A Fine-Grained World Model for Robot Manipulationcited by 95
  31. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 92
  32. 2024Video as the New Language for Real-World Decision Makingcited by 87
  33. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  34. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agentscited by 84
  35. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  36. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 69
  37. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  38. 2023Learning to Model the World with Languagecited by 68

50 most cited of 218 All 218 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024The Conversation is the Command: Interacting with Real-World Autonomous Robot Through Natural Language
  2. 2022Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformers
  3. 2023Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
  4. 2024Multimodal Human-Autonomous Agents Interaction Using Pre-Trained Language and Visual Foundation Models
  5. 2024Generative Expressive Robot Behaviors using Large Language Models
  6. 2024Yell At Your Robot: Improving On-the-Fly from Language Corrections
  7. 2024SInViG: A Self-Evolving Interactive Visual Agent for Human-Robot Interaction
  8. 2025NatSGLD: A Dataset with Speech, Gesture, Logic, and Demonstration for Robot Learning in Natural Human-Robot Interaction