Perceiving, Reasoning, Adapting: A Dual-Layer Framework for VLM-Guided Precision Robotic Manipulation
On the map: LLM reasoning → Spatial and embodied reasoning
Abstract (arXiv)
Vision-Language Models (VLMs) demonstrate remarkable potential in robotic manipulation, yet challenges persist in executing complex fine manipulation tasks with high speed and precision. While excelling at high-level planning, existing VLM methods struggle to guide robots through precise sequences of fine motor actions. To address this limitation, we introduce a progressive VLM planning algorithm that empowers robots to perform fast, precise, and error-correctable fine manipulation. Our method decomposes complex tasks into sub-actions and maintains three key data structures: task memory structure, 2D topology graphs, and 3D spatial networks, achieving high-precision spatial-semantic fusion. These three components collectively accumulate and store critical information throughout task execution, providing rich context for our task-oriented VLM interaction mechanism. This enables VLMs to dynamically adjust guidance based on real-time feedback, generating precise action plans and facilitating step-wise error correction. Experimental validation on complex assembly tasks demonstrates that our algorithm effectively guides robots to rapidly and precisely accomplish fine manipulation in challenging scenarios, significantly advancing robot intelligence for precision tasks.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 21 references that are in the map. First posted on arXiv 7 Mar 2025.
Builds on
19 of 21 references.
Closest ancestors · 2
Same direction, newest first
- 2024RT-Grasp: Reasoning Tuning Robotic Grasping via Multi-modal Large Language Model7 Nov
- 2024GameVLM: A Decision-making Framework for Robotic Task Planning Based on Visual Language Models and Zero-sum Games22 May
Other references · 17
- 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
- 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
- 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
- 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
- 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
- 2024RT-H: Action Hierarchies Using Languagecited by 204
- 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
- 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
- 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61
- 2025SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulationcited by 58
- 2024A Survey on LLM-based Multi-Agent System: Recent Advances and New Frontiers in Applicationcited by 54
- 2024VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Modelscited by 46
5 more
- 2023CoPAL: Corrective Planning of Robot Actions with Large Language Modelscited by 39
- 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
- 2024A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulationcited by 20
- 2025TRG-planner: Traversal Risk Graph-Based Path Planning in Unstructured Environments for Safe and Efficient Navigationcited by 5
- 2023Topo-Geometrically Distinct Path Computation using Neighborhood-augmented Graph, and its Application to Path Planning for a Tethered Robot in 3Dcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey
- 2026DAM-VLA: A Dynamic Action Model-Based Vision-Language-Action Framework for Robot Manipulation
- 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulation
- 2025Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation
- 2025Unified Embodied VLM Reasoning with Robotic Action via Autoregressive Discretized Pre-training
- 2025Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulation
- 2025Towards Logic-Aware Manipulation: A Knowledge Primitive for VLM-Based Assistants in Smart Manufacturing
- 2025Rethinking Intermediate Representation for VLM-based Robot Manipulation