Towards the Harness of Embodied Agents
On the map: LLM agents and tool use → Multimodal and embodied agents
Abstract (arXiv)
The success of coding agents has established the harness as a paradigm: what an agent achieves depends not on the model alone, but on the infrastructure around it. We ask whether the same paradigm extends to embodied agents in the physical world. We present Thea, a harness in which an agentic loop orchestrates robot capabilities, each wrapped as a callable tool. It inherits the core components of coding agents, modified as the physical world requires. The world, however, withholds two abilities that software grants for free: reading the state of the world, and judging the outcome of an action. To bridge these gaps, Thea introduces Scene Graph as Context, a persistent, symbolic representation of the world, and Evaluation as Exit Codes, which detects when an action should terminate, judges whether it succeeded, and on failure diagnoses the cause. Together they close the loop between the agent and the physical world. Rich behaviors then emerge from the composition of tools, and the closed loop carries long-horizon tasks to completion in real environments.
Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 42 of its 55 references that are in the map. First posted on arXiv 3 Aug 2026.
Builds on
42 of 55 references.
Closest ancestors · 5
Same direction, newest first
- 2026ASPIRE: Agentic /Skills Discovery for Robotics30 Jun
- 2026ENPIRE: Agentic Robot Policy Self-Improvement in the Real World18 Jun
- 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation23 Mar
- 2026RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks12 Mar
- 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents23 Jan
Other references · 37
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
- 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
- 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
- 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
25 more
- 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
- 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
- 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
- 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
- 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
- 2026World Action Models are Zero-shot Policiescited by 411
- 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
- 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
- 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
- 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
- 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
- 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
- 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 141
- 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulationcited by 118
- 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 58
- 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
- 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactionscited by 46
- 2023Statler: State-Maintaining Language Models for Embodied Reasoningcited by 40
- 2026From Foundation to Application: Improving VLA Models in Practicecited by 25
- 2026Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harnesscited by 7
- 2026SysNav: Multi-Level Systematic Cooperation Enables Real-World, Cross-Embodiment Object Navigationcited by 4
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026Task-Specified Active Metrological Inspection with Measurement-Steered VLA Manipulation and Deterministic Evidence Gatingcited by 0
- 2026AeroWeaver: An Embodied-Agent Harness for Weaving Aerial Skills into Distributed, Adaptive Swarm Executioncited by 0
- 2026MaskHarness-WAM: Instance-Grounded Harnessing for Long-Horizon Robot Manipulationcited by 0
- 2026ME-VLM: A Unified VLM for Embodied Cognition and Agent Coordinationcited by 0
- 2026Self-Evolving Coding Agents: From Digital Programs to Physical-World Intelligencecited by 0
- 2026Spotter: Let the Embodied Model Lead, and the VLM Reflect for Itcited by 0
- 2026DynaHarness: A Dynamic Physical Harness for Self-Evolving Robot Agentscited by 0
- 2026DynaHarness: A Dynamic Physical Harness for Self-Evolving Robot Agents30 Sep 2026
- 2026Spotter: Let the Embodied Model Lead, and the VLM Reflect for It29 Sep 2026
- 2026Self-Evolving Coding Agents: From Digital Programs to Physical-World Intelligence28 Sep 2026
- 2026ME-VLM: A Unified VLM for Embodied Cognition and Agent Coordination21 Sep 2026
- 2026MaskHarness-WAM: Instance-Grounded Harnessing for Long-Horizon Robot Manipulation17 Sep 2026
- 2026AeroWeaver: An Embodied-Agent Harness for Weaving Aerial Skills into Distributed, Adaptive Swarm Execution16 Sep 2026
- 2026Task-Specified Active Metrological Inspection with Measurement-Steered VLA Manipulation and Deterministic Evidence Gating13 Sep 2026
Similar, not cited
Close in wording, no citation either way
- 2026Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation
- 2026ETA: A New Agentic Paradigm for Embodied Tasks
- 2026HarnessPAI: An Evolving Harness for Physical AI
- 2026ASENA: Self-evolving Agents for Embodied Navigation
- 2026Compiling and Benchmarking Task-State Horizons for Embodied Agents
- 2026Zetta zeta: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence
- 2026From Digital to Physical: Digital Agents as Autonomous Coaches for Physical Intelligence
- 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses