mlsanityData to 7 Oct 2026

Diagnosing Semantic Handoff Failures in Agent-Orchestrated Vision-Language-Action Skill Composition

arXiv 2607.06256 ↗First posted 7 Jul 2026cs.ROcited by 1

On the map: Vision-language-action models → Failure recovery and robustness

also in 1 more
Abstract (arXiv)

Long-horizon household tasks require robots to compose many language-conditioned skills, yet the boundary between consecutive skills is rarely explicit. A skill may satisfy its own postcondition while leaving the robot, objects, or camera views in a state from which the next skill cannot reliably start. We study this semantic handoff problem in BEHAVIOR-1K through an agent-orchestrated vision-language-action execution harness. The harness invokes π0.5-based skill checkpoints trained from cleaned BEHAVIOR-1K demonstrations, assigns each skill typed arguments and a step budget, and uses multi-view vision-language model verification to decide whether execution should advance, retry, or replan. To separate isolated skill competence from long-horizon compositional robustness, we evaluate the same checkpoints under two initial-state distributions: clean skill-boundary snapshots and chained terminal states produced by previous skills. Selected navigation, grasping, placement, and door-opening skills achieve 77--100% success from clean snapshots under human-reviewed verification, yet composed rollouts still frequently stall from chained states. The resulting traces attribute failures to next-skill readiness, target grounding, and control execution, turning nearzero task success into actionable diagnostics for what VLA skill libraries must learn next: robustness to the messy chained-state distribution that clean demonstrations underrepresent.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Failure recovery and robustness, one of 13 directions in Vision-language-action models. Builds on 26 of its 31 references that are in the map. First posted on arXiv 7 Jul 2026.

Builds on

26 of 31 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation23 Mar
  2. 2026RoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks12 Mar
  3. 2025ManiAgent: An Agentic Framework for General Robotic Manipulation13 Oct

Other references · 23

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  9. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  11. 2022A Generalist Agentcited by 802
  12. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
11 more
  1. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  2. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  3. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  4. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 351
  5. 2025MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulationcited by 183
  6. 2018PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planningcited by 147
  7. 2025RoboBrain 2.0 Technical Reportcited by 87
  8. 2025Agentic Robot: A Brain-Inspired Framework for Vision-Language-Action Models in Embodied Agentscited by 30
  9. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  10. 2025An Atomic Skill Library Construction Method for Data-Efficient Embodied Manipulationcited by 10
  11. 2025Maestro: Orchestrating Robotics Modules with Vision-Language Models for Zero-Shot Generalist Robotscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Don't Drop the BATON: Long-Horizon Robot Manipulation via Agentic Subtask Exploration and Transition-aware Memorycited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2026Behavior-Skill: A Fine-Grained Benchmark for Evaluating Vision-Language-Action Policies in Long-Horizon Tasks
  2. 2026LIBERO-VIFO: Benchmarking the Capability and Safety of Visual Cue Following in Vision-Language-Action Models
  3. 2026RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models
  4. 2025SwitchVLA: Execution-Aware Task Switching for Vision-Language-Action Models
  5. 2026InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation
  6. 2026Unmasking the Illusion of Embodied Reasoning in Vision-Language-Action Models
  7. 2026When Instructions Retrieve Trajectories: Diagnosing and Mitigating Generalization Failures in VLA Models
  8. 2026Scaling Verification Can Be More Effective than Scaling Policy Learning for Vision-Language-Action Alignment