# From Reaction to Anticipation: Proactive Failure Recovery through Agentic Task Graph for Robotic Manipulation arXiv: https://arxiv.org/abs/2605.11951 ยท HTML: https://mlsanity.com/paper/2605.11951/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. - Builds on 38 of its 61 references that are in the map. - First posted on arXiv 12 May 2026. ## Placed in - LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/) ## Builds on (most cited first, 30 of 38) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [GR00T N1: An Open Foundation Model for Generalist Humanoid Robots](https://mlsanity.com/paper/2503.14734/) (2025) - [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022) - [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022) - [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025) - [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025) - [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023) - [RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation](https://mlsanity.com/paper/2410.07864/) (2024) - [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022) - [RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation](https://mlsanity.com/paper/2506.18088/) (2025) - [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023) - [Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents](https://mlsanity.com/paper/2302.01560/) (2023) - [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024) - [Foundation Models in Robotics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2312.07843/) (2023) - [AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains](https://mlsanity.com/paper/2212.08333/) (2022) - [REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction](https://mlsanity.com/paper/2306.15724/) (2023) - [Vision-Language Models as Success Detectors](https://mlsanity.com/paper/2303.07280/) (2023) - [AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation](https://mlsanity.com/paper/2410.00371/) (2024) - [Manipulate-Anything: Automating Real-World Robots using Vision-Language Models](https://mlsanity.com/paper/2406.18915/) (2024) - [Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection](https://mlsanity.com/paper/2412.04455/) (2024) - [DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment](https://mlsanity.com/paper/2307.00329/) (2023) - [Real-World Robot Applications of Foundation Models: A Review](https://mlsanity.com/paper/2402.05741/) (2024) - [RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning](https://mlsanity.com/paper/2409.14674/) (2024) - [Challenges and Outlook in Robotic Manipulation of Deformable Objects](https://mlsanity.com/paper/2105.01767/) (2021) - [RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction](https://mlsanity.com/paper/2505.12224/) (2025) - [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024) - [Towards a Unified Understanding of Robot Manipulation: A Comprehensive Survey](https://mlsanity.com/paper/2510.10903/) (2025) - [You Only Teach Once: Learn One-Shot Bimanual Robotic Manipulation from Video Demonstrations](https://mlsanity.com/paper/2501.14208/) (2025) ## Built on by (most cited first, 7 of 7) - [FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvement](https://mlsanity.com/paper/2607.01111/) (2026) - [DualManip: Agentic Dynamic Manipulation via Dual-Path Semantic Reasoning and Geometric Adaptation](https://mlsanity.com/paper/2609.31112/) (2026) - [In-Context Learning for Robots: Methods and Applications](https://mlsanity.com/paper/2609.36012/) (2026) - [Decoupling Intention from Trajectory: A Representational Deduction Framework for World Action Models](https://mlsanity.com/paper/2608.06994/) (2026) - [Explicit State Elicitation Is Not Enough: A Controlled Audit of Memory-Policy Classification](https://mlsanity.com/paper/2608.17247/) (2026) - [Noise Floor Audit for Agent Benchmarks](https://mlsanity.com/paper/2608.22331/) (2026) - [HaWMPO: Hallucination-Aware World Model-based Policy Optimization for Generalist Robot Policy](https://mlsanity.com/paper/2609.09941/) (2026) ## Abstract (arXiv) Although robotic manipulation has made significant progress, reliable execution remains challenging because task failures are inevitable in dynamic and unstructured environments. To handle such failures, existing frameworks typically follow a stepwise detect-reason-recover pipeline, which often incurs high latency and limited robustness due to delayed reasoning and reactive planning. Inspired by the human capability to anticipate and proactively plan for potential failures, we introduce AgentChord, an agentic system that models a manipulation task as a directed task graph. Before execution, this graph is enriched with anticipatory recovery branches that specify context-aware corrective behaviors, enabling immediate and targeted responses when failures occur. Specifically, AgentChord operates through a choreography of specialized agents: a composer that structures the nominal task graph, an arranger that augments the graph with anticipatory recovery branches, and a conductor that compiles and coordinates executable transitions using low-latency monitors to detect deviations and trigger pre-compiled recoveries without re-planning. Empirical studies on diverse long-horizon bimanual manipulation tasks demonstrate that AgentChord substantially improves success rates and execution efficiency, advancing the reliability and autonomy of real-world robotic systems. The project page is available at: https://shengxu.net/AgentChord/.