# From Perception to Symbolic Task Planning: Vision-Language Guided Human-Robot Collaborative Structured Assembly

arXiv: https://arxiv.org/abs/2601.00978 · HTML: https://mlsanity.com/paper/2601.00978/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jan 2026.

## Built on by (most cited first, 2 of 2)

- [Constraint-Grounded Reinforcement Learning for Variable Impedance Control in Contact-Rich Robotic Insertion](https://mlsanity.com/paper/2609.13516/) (2026)
- [From Documented Strengths to Force Limits: Material-Informed Robotic Insertion for Construction Assembly](https://mlsanity.com/paper/2609.22609/) (2026)

## Abstract (arXiv)

Human-robot collaboration (HRC) in structured assembly requires reliable state estimation and adaptive task planning under noisy perception and human interventions. To address these challenges, we introduce a design-grounded human-aware planning framework for human-robot collaborative structured assembly. The framework comprises two coupled modules. Module I, Perception-to-Symbolic State (PSS), employs vision-language models (VLMs) based agents to align RGB-D observations with design specifications and domain knowledge, synthesizing verifiable symbolic assembly states. It outputs validated installed and uninstalled component sets for online state tracking. Module II, Human-Aware Planning and Replanning (HPR), performs task-level multi-robot assignment and updates the plan only when the observed state deviates from the expected execution outcome. It applies a minimal-change replanning rule to selectively revise task assignments and preserve plan stability even under human interventions. We validate the framework on a 27-component timber-frame assembly. The PSS module achieves 97% state synthesis accuracy, and the HPR module maintains feasible task progression across diverse HRC scenarios. Results indicate that integrating VLM-based perception with knowledge-driven planning improves robustness of state estimation and task planning under dynamic conditions.