mlsanityData to 7 Oct 2026

LILA: Language-Informed Latent Actions

arXiv 2111.03205 ↗First posted 5 Nov 2021cs.RO cs.AI cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

We introduce Language-Informed Latent Actions (LILA), a framework for learning natural language interfaces in the context of human-robot collaboration. LILA falls under the shared autonomy paradigm: in addition to providing discrete language inputs, humans are given a low-dimensional controller - e.g., a 2 degree-of-freedom (DoF) joystick that can move left/right and up/down - for operating the robot. LILA learns to use language to modulate this controller, providing users with a language-informed control space: given an instruction like "place the cereal bowl on the tray," LILA may learn a 2-DoF space where one dimension controls the distance from the robot's end-effector to the bowl, and the other dimension controls the robot's end-effector pose relative to the grasp point on the bowl. We evaluate LILA with real-world user studies, where users can provide a language instruction while operating a 7-DoF Franka Emika Panda Arm to complete a series of complex manipulation tasks. We show that LILA models are not only more sample efficient and performant than imitation learning and end-effector control baselines, but that they are also qualitatively preferred by users.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 38 of its 63 references that are in the map. First posted on arXiv 5 Nov 2021.

Builds on

38 of 63 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  7. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  8. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  9. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  10. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  11. 2016Tutorial on Variational Autoencoderscited by 404
  12. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
26 more
  1. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  2. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  3. 2017DART: Noise Injection for Robust Imitation Learningcited by 187
  4. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  5. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  6. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  7. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  8. 2017Gated-Attention Architectures for Task-Oriented Language Groundingcited by 72
  9. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
  10. 2018Shared Autonomy via Deep Reinforcement Learningcited by 65
  11. 2020Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Textcited by 59
  12. 2017Shared Autonomy via Hindsight Optimization for Teleoperation and Teamingcited by 39
  13. 2018Guiding Policies with Language via Meta-Learningcited by 34
  14. 2020Shared Autonomy with Learned Latent Actionscited by 34
  15. 2019Controlling Assistive Robots with Learned Latent Actionscited by 28
  16. 2018Following High-level Navigation Instructions on a Simulated Quadcopter with Imitation Learningcited by 27
  17. 2020Residual Policy Learning for Shared Autonomycited by 25
  18. 2020Learning Adaptive Language Interfaces through Decompositioncited by 22
  19. 2017Naturalizing a Programming Language via Interactive Learningcited by 20
  20. 2017Accurately and Efficiently Interpreting Human-Robot Instructions of Varying Granularitiescited by 19
  21. 2021Learning Latent Actions to Control Assistive Robotscited by 17
  22. 2020Unnatural Language Processing: Bridging the Gap Between Synthetic and Natural Language Datacited by 16
  23. 2021Learning Visually Guided Latent Actions for Assistive Teleoperationcited by 12
  24. 2018HARMONIC: A Multimodal Dataset of Assistive Human-Robot Collaborationcited by 10
  25. 2017A Tale of Two DRAGGNs: A Hybrid Approach for Interpreting Action-Oriented and Goal-Oriented Instructionscited by 4
  26. 2020Learning User-Preferred Mappings for Intuitive Robot Controlcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 7; 2024: 8; 2025: 6; 2026: 1.

27 papers.

Sort:

By area:

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  3. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  4. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  5. 2023Language-Driven Representation Learning for Roboticscited by 163
  6. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  7. 2022Correcting Robot Plans with Natural Language Feedbackcited by 68
  8. 2023"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomycited by 60
  9. 2024RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learningcited by 46
  10. 2024True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learningcited by 36
  11. 2023LLF-Bench: Benchmark for Interactive Learning from Language Feedbackcited by 30
  12. 2024Natural Language Can Help Bridge the Sim2Real Gapcited by 21
15 more
  1. 2024Explaining Datasets in Words: Statistical Models with Natural Language Parameterscited by 21
  2. 2022Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Taskscited by 17
  3. 2022Assistive Teaching of Motor Control Tasks to Humanscited by 8
  4. 2024Dreaming to Assist: Learning to Align with Human Objectives for Shared Control in High-Speed Racingcited by 7
  5. 2023Generating Language Corrections for Teaching Physical Control Taskscited by 4
  6. 2024Conformalized Teleoperation: Confidently Mapping Human Inputs to High-Dimensional Robot Actionscited by 4
  7. 2024LIMT: Language-Informed Multi-Task Visual World Modelscited by 4
  8. 2024Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaborationcited by 4
  9. 2025Shared Autonomy for Proximal Teachingcited by 4
  10. 2025LAMS: LLM-Driven Automatic Mode Switching for Assistive Teleoperationcited by 2
  11. 2025Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Modelscited by 1
  12. 2025Mixed-Initiative Dialog for Human-Robot Collaborative Manipulationcited by 1
  13. 2025A Signal Contract for Online Language Grounding and Discovery in Decision-Makingcited by 0
  14. 2025QuickLAP: Quick Language-Action Preference Learning for Semi-Autonomous Agentscited by 0
  15. 2026Steering Autoregressive Vision-Language-Action Policies via Action Token Interventioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
  2. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks
  3. 2025LACY: A Vision-Language Model-based Language-Action Cycle for Self-Improving Robotic Manipulation
  4. 2025CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguity
  5. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models
  6. 2026LiLa-WAM: Lightweight Latent Reasoning World-Action Model for Robotic Manipulation
  7. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfer
  8. 2025LaGEA: Language Guided Embodied Agents for Robotic Manipulation