mlsanityData to 7 Oct 2026

Towards the Harness of Embodied Agents

arXiv 2608.11246 ↗First posted 3 Aug 2026cs.AI cs.LG cs.ROcited by 7

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

The success of coding agents has established the harness as a paradigm: what an agent achieves depends not on the model alone, but on the infrastructure around it. We ask whether the same paradigm extends to embodied agents in the physical world. We present Thea, a harness in which an agentic loop orchestrates robot capabilities, each wrapped as a callable tool. It inherits the core components of coding agents, modified as the physical world requires. The world, however, withholds two abilities that software grants for free: reading the state of the world, and judging the outcome of an action. To bridge these gaps, Thea introduces Scene Graph as Context, a persistent, symbolic representation of the world, and Evaluation as Exit Codes, which detects when an action should terminate, judges whether it succeeded, and on failure diagnoses the cause. Together they close the loop between the agent and the physical world. Rich behaviors then emerge from the composition of tools, and the closed loop carries long-horizon tasks to completion in real environments.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 42 of its 55 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

42 of 55 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026ASPIRE: Agentic /Skills Discovery for Robotics30 Jun
  2. 2026ENPIRE: Agentic Robot Policy Self-Improvement in the Real World18 Jun
  3. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation23 Mar
  4. 2026RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks12 Mar
  5. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents23 Jan

Other references · 37

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  6. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  11. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  12. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
25 more
  1. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  3. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  5. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  6. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  7. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  8. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  9. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  10. 2026World Action Models are Zero-shot Policiescited by 411
  11. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 357
  12. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  13. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  14. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  15. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  16. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  17. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 141
  18. 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulationcited by 118
  19. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 58
  20. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  21. 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactionscited by 46
  22. 2023Statler: State-Maintaining Language Models for Embodied Reasoningcited by 40
  23. 2026From Foundation to Application: Improving VLA Models in Practicecited by 25
  24. 2026Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harnesscited by 7
  25. 2026SysNav: Multi-Level Systematic Cooperation Enables Real-World, Cross-Embodiment Object Navigationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation
  2. 2026ETA: A New Agentic Paradigm for Embodied Tasks
  3. 2026HarnessPAI: An Evolving Harness for Physical AI
  4. 2026ASENA: Self-evolving Agents for Embodied Navigation
  5. 2026Compiling and Benchmarking Task-State Horizons for Embodied Agents
  6. 2026Zetta zeta: An Efficient Closed-Loop Embodied Harness for Self-Evolving Physical Intelligence
  7. 2026From Digital to Physical: Digital Agents as Autonomous Coaches for Physical Intelligence
  8. 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses