mlsanityData to 7 Oct 2026

Lifelong Robot Learning with Human Assisted Language Planners

arXiv 2309.14321 ↗First posted 25 Sep 2023cs.RO cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have been shown to act like planners that can decompose high-level instructions into a sequence of executable instructions. However, current LLM-based planners are only able to operate with a fixed set of skills. We overcome this critical limitation and present a method for using LLM-based planners to query new skills and teach robots these skills in a data and time-efficient manner for rigid object manipulation. Our system can re-use newly acquired skills for future tasks, demonstrating the potential of open world and lifelong learning. We evaluate the proposed framework on multiple tasks in simulation and the real world. Videos are available at: https://sites.google.com/mit.edu/halp-robot-learning.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 35 of its 42 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

35 of 42 references.

References · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Segment Anythingcited by 8,158
  6. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  7. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  12. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
23 more
  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  3. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  4. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  5. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  6. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  7. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  8. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  10. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  11. 2021Implicit Behavioral Cloningcited by 442
  12. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  13. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  14. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  15. 2020Integrated Task and Motion Planningcited by 281
  16. 2017One-Shot Imitation Learningcited by 232
  17. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  18. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  19. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  20. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 204
  21. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  22. 2018Zero-Shot Visual Imitationcited by 114
  23. 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 7; 2026: 2.

19 papers.

Sort:

By area:

  1. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  2. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  3. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Makingcited by 142
  4. 2024Query2CAD: Generating CAD models using natural language queriescited by 28
  5. 2025SPECI: Skill Prompts based Hierarchical Continual Imitation Learning for Robot Manipulationcited by 9
  6. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 9
  7. 2026Uni-Skill: Building Self-Evolving Skill Repository for Generalizable Robotic Manipulationcited by 8
  8. 2024Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Usecited by 7
  9. 2024TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Modelscited by 4
  10. 2025OpenNav: Open-World Navigation with Multimodal Large Language Modelscited by 3
  11. 2024Position Paper: Agent AI Towards a Holistic Intelligencecited by 2
  12. 2024GRAPPA: Generalizing and Adapting Robot Policies via Online Agentic Guidancecited by 2
7 more
  1. 2024Advances in Transformers for Robotic Applications: A Reviewcited by 2
  2. 2024Task-agnostic Lifelong Robot Learning with Retrieval-based Weighted Local Adaptationcited by 1
  3. 2025Prompt-responsive Object Retrieval with Memory-augmented Student-Teacher Learningcited by 1
  4. 2025CARoL: Context-aware Adaptation for Robot Learningcited by 1
  5. 2025Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Modelscited by 1
  6. 2025LLM-Driven Self-Refinement for Embodied Drone Task Planningcited by 1
  7. 2026SkillComposer: Learning Reusable Skills for Natural-Language Robot Programmingcited by 0