# A Physical Agentic Loop for Language-Guided Grasping with Execution-State Monitoring

arXiv: https://arxiv.org/abs/2604.07395 · HTML: https://mlsanity.com/paper/2604.07395/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 27 references that are in the map.
- First posted on arXiv 8 Apr 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 25 of 25)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction](https://mlsanity.com/paper/2306.15724/) (2023)
- [AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation](https://mlsanity.com/paper/2410.00371/) (2024)
- [Manipulate-Anything: Automating Real-World Robots using Vision-Language Models](https://mlsanity.com/paper/2406.18915/) (2024)
- [Correcting Robot Plans with Natural Language Feedback](https://mlsanity.com/paper/2204.05186/) (2022)
- ["No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy](https://mlsanity.com/paper/2301.02555/) (2023)
- [Agentic Robot: A Brain-Inspired Framework for Vision-Language-Action Models in Embodied Agents](https://mlsanity.com/paper/2505.23450/) (2025)
- [Slip Detection with Combined Tactile and Visual Information](https://mlsanity.com/paper/1802.10153/) (2018)
- [FPC-VLA: A Vision-Language-Action Framework with a Supervisor for Failure Prediction and Correction](https://mlsanity.com/paper/2509.04018/) (2025)
- [Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols](https://mlsanity.com/paper/2512.02787/) (2025)
- [I-FailSense: Towards General Robotic Failure Detection with Vision-Language Models](https://mlsanity.com/paper/2509.16072/) (2025)
- [ForceSight: Text-Guided Mobile Manipulation with Visual-Force Goals](https://mlsanity.com/paper/2309.12312/) (2023)
- [ManiAgent: An Agentic Framework for General Robotic Manipulation](https://mlsanity.com/paper/2510.11660/) (2025)
- [HiCRISP: An LLM-based Hierarchical Closed-Loop Robotic Intelligent Self-Correction Planner](https://mlsanity.com/paper/2309.12089/) (2023)
- [ARRC: Advanced Reasoning Robot Control - Knowledge-Driven Autonomous Manipulation Using Retrieval-Augmented Generation](https://mlsanity.com/paper/2510.05547/) (2025)
- [VCoT-Grasp: Grasp Foundation Models with Visual Chain-of-Thought Reasoning for Language-driven Grasp Generation](https://mlsanity.com/paper/2510.05827/) (2025)
- [Sim2Real Transfer for Vision-Based Grasp Verification](https://mlsanity.com/paper/2505.03046/) (2025)

## Abstract (arXiv)

Robotic manipulation systems that follow language instructions often execute grasp primitives in a largely single-shot manner: a model proposes an action, the robot executes it, and failures such as empty grasps, slips, stalls, timeouts, or semantically wrong grasps are not surfaced to the decision layer in a structured way. Inspired by agentic loops in digital tool-using agents, we reformulate language-guided grasping as a bounded embodied agent operating over grounded execution states, where physical actions expose an explicit tool-state stream. We introduce a physical agentic loop that wraps an unmodified learned manipulation primitive (grasp-and-lift) with (i) an event-based interface and (ii) an execution monitoring layer, Watchdog, which converts noisy gripper telemetry into discrete outcome labels using contact-aware fusion and temporal stabilization. These outcome events, optionally combined with post-grasp semantic verification, are consumed by a deterministic bounded policy that finalizes, retries, or escalates to the user for clarification, guaranteeing finite termination. We validate the resulting loop on a mobile manipulator with an eye-in-hand D405 camera, keeping the underlying grasp model unchanged and evaluating representative scenarios involving visual ambiguity, distractors, and induced execution failures. Results show that explicit execution-state monitoring and bounded recovery enable more robust and interpretable behavior than open-loop execution, while adding minimal architectural overhead. For the source code and demo refer to our project page: https://wenzewwz123.github.io/Agentic-Loop/