mlsanityData to 7 Oct 2026

MistyPilot: Enabling Social-Robot Control through Multi-Agent LLM Skill Orchestration

arXiv 2608.15549 ↗First posted 16 Aug 2026cs.RO cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Programming small social robots from natural-language instructions requires more than invoking isolated APIs. Interactive tasks combine reactive physical behaviors with stateful social behaviors, while existing interfaces often require developers to manually compose APIs into skills, configure their parameters, bind sensor events to skills, and manage task states at runtime. We present MistyPilot, a multi-agent LLM framework that interprets high-level natural-language instructions and orchestrates the corresponding skills on the Misty social robot. A Task Router dispatches each instruction to one of two specialized agents: a Physically Interactive Agent for sensor-triggered robot control and direct skill invocation, and a Social Interaction Agent for dialogue-oriented task-state management and context-dependent multimodal response generation. To improve efficiency, the Social Interaction Agent reuses previously generated results when applicable and invokes full generation otherwise. We evaluate MistyPilot on five component-level suites, with sensor bindings and skill invocations executed on the physical Misty robot, and a preliminary user study with 12 participants. MistyPilot attains high accuracy on routing, sensor-skill binding, task-state parsing, result reuse, and skill extension up to 100 skills, and lower variance than an otherwise identical single-agent baseline, while participants report positive perceptions of usability and interaction quality. The code will be made publicly available via the project page.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 19 of its 35 references that are in the map. First posted on arXiv 16 Aug 2026.

Builds on

19 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025AutoMisty: A Multi-Agent LLM Framework for Automated Code Generation in the Misty Social Robot9 Mar
  2. 2024Tulip Agent -- Enabling LLM-Based Agents to Solve Tasks Using Large Tool Libraries31 Jul
  3. 2024A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learning9 Jun

Other references · 16

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  6. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  7. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  8. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  9. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  10. 2024Tool Learning with Large Language Models: A Surveycited by 221
  11. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  12. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
4 more
  1. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agentscited by 83
  2. 2024What Are Tools Anyway? A Survey from the Language Model Perspectivecited by 53
  3. 2024Odyssey: Empowering Minecraft Agents with Open-World Skillscited by 26
  4. 2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 18

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MistyPilot: An Agentic Fast-Slow Thinking LLM Framework for Misty Social Robots
  2. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models
  3. 2025Building Knowledge from Interactions: An LLM-Based Architecture for Adaptive Tutoring and Social Reasoning
  4. 2026Ludiscriptscriptstyle 0.1: An Agentic System for Socially Intelligent Robots
  5. 2024ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning
  6. 2025MinionsLLM: a Task-adaptive Framework For The Training and Control of Multi-Agent Systems Through Natural Language
  7. 2026ARIS: Agentic and Relationship Intelligence System for Social Robots
  8. 2026A Multimodal Framework for Human-Multi-Agent Interaction