# Enabling Novel Mission Operations and Interactions with ROSA: The Robot Operating System Agent arXiv: https://arxiv.org/abs/2410.06472 ยท HTML: https://mlsanity.com/paper/2410.06472/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. - Builds on 13 of its 47 references that are in the map. - First posted on arXiv 9 Oct 2024. ## Placed in - LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/) ## Builds on (most cited first, 13 of 13) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022) - [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023) - [Building Cooperative Embodied Agents Modularly with Large Language Models](https://mlsanity.com/paper/2307.02485/) (2023) - [Robot Operating System 2: Design, Architecture, and Uses In The Wild](https://mlsanity.com/paper/2211.07752/) (2022) - [An Introduction to Vision-Language Modeling](https://mlsanity.com/paper/2405.17247/) (2024) - [Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification](https://mlsanity.com/paper/2406.08660/) (2024) - [LATTE: LAnguage Trajectory TransformEr](https://mlsanity.com/paper/2208.02918/) (2022) - [Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectives](https://mlsanity.com/paper/2407.14962/) (2024) - [HRI Challenges Influencing Low Usage of Robotic Systems in Disaster Response and Rescue Operations](https://mlsanity.com/paper/2401.15760/) (2024) - [Copiloting Autonomous Multi-Robot Missions: A Game-inspired Supervisory Control Interface](https://mlsanity.com/paper/2204.06647/) (2022) ## Built on by (most cited first, 16 of 16) - [Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interaction](https://mlsanity.com/paper/2508.05294/) (2025) - [AEROS: A Single-Agent Operating Architecture with Embodied Capability Modules](https://mlsanity.com/paper/2604.07039/) (2026) - [RAI: Flexible Agent Framework for Embodied AI](https://mlsanity.com/paper/2505.07532/) (2025) - [Taking Flight with Dialogue: Enabling Natural Language Control for PX4-based Drone Agent](https://mlsanity.com/paper/2506.07509/) (2025) - [Distributed AI Agents for Cognitive Underwater Robot Autonomy](https://mlsanity.com/paper/2507.23735/) (2025) - [ROSClaw: An OpenClaw ROS 2 Framework for Agentic Robot Control and Interaction](https://mlsanity.com/paper/2603.26997/) (2026) - [Agentic Scene Policies: Unifying Space, Semantics, and Affordances for Robot Action](https://mlsanity.com/paper/2509.19571/) (2025) - [ROSBag MCP Server: Analyzing Robot Data with LLMs for Agentic Embodied AI Applications](https://mlsanity.com/paper/2511.03497/) (2025) - [A ROS 2 Wrapper for Florence-2: Multi-Mode Local Vision-Language Inference for Robotic Systems](https://mlsanity.com/paper/2604.01179/) (2026) - [Improving Cooperation in Collaborative Embodied AI](https://mlsanity.com/paper/2510.03153/) (2025) - [EmbodiedLGR: Integrating Lightweight Graph Representation and Retrieval for Semantic-Spatial Memory in Robotic Agents](https://mlsanity.com/paper/2604.18271/) (2026) - [Octopus Protocol: One-Shot Hardware Discovery and Control for AI Agents via Infrastructure-as-Prompts](https://mlsanity.com/paper/2605.09055/) (2026) - [SPINE: Bridging the Cyber-Physical Gap with Agentic AI](https://mlsanity.com/paper/2607.13049/) (2026) - [Bridging Thought and Action: Taming Long-Horizon Instability in Open-Source LLM Agents with a MetaTool-Enhanced ROS Framework](https://mlsanity.com/paper/2609.13335/) (2026) - [Spatial and Semantic Reasoning for LLM-Driven Robot Navigation via MCP](https://mlsanity.com/paper/2609.27340/) (2026) - [CognitiveReality: Robot-Agnostic Semantic Gaussian Mapping with an LLM Agent for Immersive Collaborative VR Teleoperation](https://mlsanity.com/paper/2609.31418/) (2026) ## Abstract (arXiv) The advancement of robotic systems has revolutionized numerous industries, yet their operation often demands specialized technical knowledge, limiting accessibility for non-expert users. This paper introduces ROSA (Robot Operating System Agent), an AI-powered agent that bridges the gap between the Robot Operating System (ROS) and natural language interfaces. By leveraging state-of-the-art language models and integrating open-source frameworks, ROSA enables operators to interact with robots using natural language, translating commands into actions and interfacing with ROS through well-defined tools. ROSA's design is modular and extensible, offering seamless integration with both ROS1 and ROS2, along with safety mechanisms like parameter validation and constraint enforcement to ensure secure, reliable operations. While ROSA is originally designed for ROS, it can be extended to work with other robotics middle-wares to maximize compatibility across missions. ROSA enhances human-robot interaction by democratizing access to complex robotic systems, empowering users of all expertise levels with multi-modal capabilities such as speech integration and visual perception. Ethical considerations are thoroughly addressed, guided by foundational principles like Asimov's Three Laws of Robotics, ensuring that AI integration promotes safety, transparency, privacy, and accountability. By making robotic technology more user-friendly and accessible, ROSA not only improves operational efficiency but also sets a new standard for responsible AI use in robotics and potentially future mission operations. This paper introduces ROSA's architecture and showcases initial mock-up operations in JPL's Mars Yard, a laboratory, and a simulation using three different robots. The core ROSA library is available as open-source.