mlsanityData to 7 Oct 2026

ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning

arXiv 2406.19741 ↗First posted 28 Jun 2024cs.RO cs.AIcited by 27

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

We present a framework for intuitive robot programming by non-experts, leveraging natural language prompts and contextual information from the Robot Operating System (ROS). Our system integrates large language models (LLMs), enabling non-experts to articulate task requirements to the system through a chat interface. Key features of the framework include: integration of ROS with an AI agent connected to a plethora of open-source and commercial LLMs, automatic extraction of a behavior from the LLM output and execution of ROS actions/services, support for three behavior modes (sequence, behavior tree, state machine), imitation learning for adding new robot actions to the library of possible actions, and LLM reflection via human and environment feedback. Extensive experiments validate the framework, showcasing robustness, scalability, and versatility in diverse scenarios, including long-horizon tasks, tabletop rearrangements, and remote supervisory control. To facilitate the adoption of our framework and support the reproduction of our results, we have made our code open-source. You can access it at: https://github.com/huawei-noah/HEBO/tree/master/ROSLLM.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 55 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

29 of 55 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMs20 Mar
  2. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought24 May

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  8. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  10. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  11. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  12. 2022Robot Operating System 2: Design, Architecture, and Uses In The Wildcited by 190
15 more
  1. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  2. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 123
  3. 2023Chain of Code: Reasoning with a Language Model-Augmented Code Emulatorcited by 120
  4. 2014Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Studycited by 86
  5. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 66
  6. 2023HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attributioncited by 55
  7. 2020A Survey of Behavior Trees in Robotics and AIcited by 53
  8. 2024Understanding Large-Language Model (LLM)-powered Human-Robot Interactioncited by 43
  9. 2024InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task Planningcited by 31
  10. 2023Robot Behavior-Tree-Based Task Generation with Large Language Modelscited by 22
  11. 2024LaMI: Large Language Models for Multi-Modal Human-Robot Interactioncited by 21
  12. 2021Physical Interaction as Communication: Learning Robot Objectives Online from Human Correctionscited by 19
  13. 2023Why Can Large Language Models Generate Correct Chain-of-Thoughts?cited by 16
  14. 2022ROS-PyBullet Interface: A Framework for Reliable Contact Simulation and Human-Robot Interactioncited by 5
  15. 2023OpTaS: An Optimization-based Task Specification Library for Trajectory Optimization and Model Predictive Controlcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 17; 2026: 9.

27 papers.

Sort:

By area:

  1. 2024Kolb-Based Experiential Learning for Generalist Agents with Human-Level Kaggle Data Science Performancecited by 22
  2. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  3. 2025AutoMisty: A Multi-Agent LLM Framework for Automated Code Generation in the Misty Social Robotcited by 5
  4. 2025Scaling Laws in Scientific Discovery with AI and Robot Scientistscited by 5
  5. 2026ROSClaw: An OpenClaw ROS 2 Framework for Agentic Robot Control and Interactioncited by 3
  6. 2025Tool-Aided Evolutionary LLM for Generative Policy Toward Efficient Resource Management in Wireless Federated Learningcited by 2
  7. 2025Leveraging LLMs for Mission Planning in Precision Agriculturecited by 2
  8. 2025One For All: LLM-based Heterogeneous Mission Planning in Precision Agriculturecited by 2
  9. 2025Ark: An Open-source Python-based Framework for Robot Learningcited by 2
  10. 2025Socio-technical aspects of Agentic AIcited by 2
  11. 2025LA-RCS: LLM-Agent-Based Robot Control Systemcited by 1
  12. 2025AURA: Autonomous Upskilling with Retrieval-Augmented Agentscited by 1
15 more
  1. 2025A Pragmatist Robot: Learning to Plan Tasks by Experiencing the Real Worldcited by 1
  2. 2025Reinforced Language Models for Sequential Decision Makingcited by 1
  3. 2025ROSBag MCP Server: Analyzing Robot Data with LLMs for Agentic Embodied AI Applicationscited by 1
  4. 2025Chat with UAV -- Human-UAV Interaction Based on Large Language Modelscited by 1
  5. 2026From Language to Action: Can LLM-Based Agents Be Used for Embodied Robot Cognition?cited by 1
  6. 2026A ROS 2 Wrapper for Florence-2: Multi-Mode Local Vision-Language Inference for Robotic Systemscited by 1
  7. 2026HoloAgent-0: A Unified Embodied Agent Framework with 3D Spatial Memorycited by 1
  8. 2026Towards general embodied intelligence: integrating large language models, knowledge bases, and reasoning capabilities to build the next generation of AI agentscited by 1
  9. 2025MARCO: Meta-Reflection with Cross-Referencing for Code Reasoningcited by 0
  10. 2025Interpretable Robot Control via Structured Behavior Trees and Large Language Modelscited by 0
  11. 2025Prompt2Craft: Generating Functional Craft Assemblies with LLMscited by 0
  12. 2026Hybrid Framework for Robotic Manipulation: Integrating Reinforcement Learning and Large Language Modelscited by 0
  13. 2026As You Wish: Mission Planning with Formal Verification using LLMs in Precision Agriculturecited by 0
  14. 2026SPINE: Bridging the Cyber-Physical Gap with Agentic AIcited by 0
  15. 2026Spatial and Semantic Reasoning for LLM-Driven Robot Navigation via MCPcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Ro-SLM: Onboard Small Language Models for Robot Task Planning and Operation Code Generation
  2. 2026Bridging Thought and Action: Taming Long-Horizon Instability in Open-Source LLM Agents with a MetaTool-Enhanced ROS Framework
  3. 2025Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interaction
  4. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models
  5. 2025LEO-RobotAgent: A General-purpose Robotic Agent for Language-driven Embodied Operator
  6. 2025Plan Verification for LLM-Based Embodied Task Completion Agents
  7. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectives
  8. 2025Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence