# From Perception to Assistance: Open-Vocabulary Shared Autonomy for Robotic Manipulation

arXiv: https://arxiv.org/abs/2607.17323 · HTML: https://mlsanity.com/paper/2607.17323/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 19 Jul 2026.

## Builds on (most cited first, 16 of 16)

- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [iSDF: Real-Time Neural Signed Distance Fields for Robot Perception](https://mlsanity.com/paper/2204.02296/) (2022)
- [A3VLM: Actionable Articulation-Aware Vision Language Model](https://mlsanity.com/paper/2406.07549/) (2024)
- [Versatile Multi-Contact Planning and Control for Legged Loco-Manipulation](https://mlsanity.com/paper/2308.09179/) (2023)
- [Learning Whole-body Manipulation for Quadrupedal Robot](https://mlsanity.com/paper/2308.16820/) (2023)
- [nvblox: GPU-Accelerated Incremental Signed Distance Field Mapping](https://mlsanity.com/paper/2311.00626/) (2023)
- [Hierarchical Optimization-based Control for Whole-body Loco-manipulation of Heavy Objects](https://mlsanity.com/paper/2311.00112/) (2023)
- [Corrective Shared Autonomy for Addressing Task Variability](https://mlsanity.com/paper/2102.07165/) (2021)
- [Online Estimation and Manipulation of Articulated Objects](https://mlsanity.com/paper/2601.01438/) (2026)
- [A Shared Autonomy Reconfigurable Control Framework for Telemanipulation of Multi-arm Systems](https://mlsanity.com/paper/2207.09813/) (2022)
- [A Vision-Based Shared-Control Teleoperation Scheme for Controlling the Robotic Arm of a Four-Legged Robot](https://mlsanity.com/paper/2508.14994/) (2025)

## Abstract (arXiv)

Teleoperating a robotic manipulator in industrial environments demands precision that camera-based interfaces alone struggle to deliver. The operator must align the end-effector with a target in clutter, under limited depth perception, and without colliding with the surrounding structures. This paper presents a shared-autonomy framework that assists the operator throughout this process. A single RGB-D camera captures the operator's arm motion and hand gestures without wearables, fiducials, or a calibration stage. The intended target is specified by a free-form text prompt, grounded by a vision-language model in the robot's gripper camera, and tracked across its onboard cameras by a promptable video-segmentation model, resulting in a grasp frame continuously separated from the obstacle map. Every commanded motion is executed by a GPU-accelerated model-predictive controller that enforces self- and environment-collision avoidance against an online volumetric reconstruction, while a potential field corrects the operator's reference toward the grounded target during the final approach. An autonomous mode can be gesture-triggered to complete the grasp on the same target without a separate perception pipeline. The framework is validated on a quadruped mobile manipulator. The interface achieves a positional RMSE of 59 mm relative to motion-capture ground truth, and the controller keeps the arm at least 18 cm from obstacles while the operator deliberately commands the arm into them by 6 cm. In an industrial valve manipulation and a pick-and-place task, the full framework succeeded in all trials, while ablating either the collision or the assistance module produced failures through complementary mechanisms, and autonomous execution succeeded in four of five trials per task.