# VISO-Grasp: Vision-Language Informed Spatial Object-centric 6-DoF Active View Planning and Grasping in Clutter and Invisibility

arXiv: https://arxiv.org/abs/2503.12609 · HTML: https://mlsanity.com/paper/2503.12609/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 48 references that are in the map.
- First posted on arXiv 16 Mar 2025.

## Builds on (most cited first, 30 of 36)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies](https://mlsanity.com/paper/2206.04670/) (2022)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [Real-World Robot Learning with Masked Visual Pre-training](https://mlsanity.com/paper/2210.03109/) (2022)
- [Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes](https://mlsanity.com/paper/2103.14127/) (2021)
- [Foundation Models in Robotics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2312.07843/) (2023)
- [PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Models](https://mlsanity.com/paper/2212.01558/) (2022)
- [Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations](https://mlsanity.com/paper/2104.01542/) (2021)
- [Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis](https://mlsanity.com/paper/2312.08782/) (2023)
- [Skill Induction and Planning with Latent Language](https://mlsanity.com/paper/2110.01517/) (2021)
- [Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter](https://mlsanity.com/paper/2101.01132/) (2021)
- [Revisiting Active Perception](https://mlsanity.com/paper/1603.02729/) (2016)
- [PointNetGPD: Detecting Grasp Configurations from Point Sets](https://mlsanity.com/paper/1809.06267/) (2018)
- [Deep Learning Approaches to Grasp Synthesis: A Review](https://mlsanity.com/paper/2207.02556/) (2022)
- [6-DOF Grasping for Target-driven Object Manipulation in Clutter](https://mlsanity.com/paper/1912.03628/) (2019)
- [GraspNeRF: Multiview-based 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRF](https://mlsanity.com/paper/2210.06575/) (2022)
- [NeU-NBV: Next Best View Planning Using Uncertainty Estimation in Image-Based Neural Rendering](https://mlsanity.com/paper/2303.01284/) (2023)
- [SMART: Self-supervised Multi-task pretrAining with contRol Transformers](https://mlsanity.com/paper/2301.09816/) (2023)
- [Getting MoRE out of Mixture of Language Model Reasoning Experts](https://mlsanity.com/paper/2305.14628/) (2023)
- [ActiveGS: Active Scene Reconstruction Using Gaussian Splatting](https://mlsanity.com/paper/2412.17769/) (2024)
- [GAMMA: Graspability-Aware Mobile MAnipulation Policy Learning based on Online Grasping Pose Fusion](https://mlsanity.com/paper/2309.15459/) (2023)
- [Reasoning Grasping via Multimodal Large Language Model](https://mlsanity.com/paper/2402.06798/) (2024)
- [ThinkGrasp: A Vision-Language System for Strategic Part Grasping in Clutter](https://mlsanity.com/paper/2407.11298/) (2024)
- [Closed-Loop Next-Best-View Planning for Target-Driven Grasping](https://mlsanity.com/paper/2207.10543/) (2022)
- [FoundationGrasp: Generalizable Task-Oriented Grasping with Foundation Models](https://mlsanity.com/paper/2404.10399/) (2024)
- [Affordance-Driven Next-Best-View Planning for Robotic Grasping](https://mlsanity.com/paper/2309.09556/) (2023)
- [Multi-View Picking: Next-best-view Reaching for Improved Grasping in Clutter](https://mlsanity.com/paper/1809.08564/) (2018)

## Built on by (most cited first, 11 of 11)

- [Diffusion Models for Robotic Manipulation: A Survey](https://mlsanity.com/paper/2504.08438/) (2025)
- [FlowCorrect: Efficient Interactive Correction of Generative Flow Policies for Robotic Manipulation](https://mlsanity.com/paper/2602.22056/) (2026)
- [Improving Robustness of Vision-Language-Action Models by Restoring Corrupted Visual Inputs](https://mlsanity.com/paper/2602.01158/) (2026)
- [Observer-Actor: Active Vision Imitation Learning with Sparse-View Gaussian Splatting](https://mlsanity.com/paper/2511.18140/) (2025)
- [Obstruction reasoning for robotic grasping](https://mlsanity.com/paper/2511.23186/) (2025)
- [Human-Interpretable Uncertainty Explanations for Point Cloud Registration](https://mlsanity.com/paper/2509.18786/) (2025)
- [SG-DOR: Learning Scene Graphs with Direction-Conditioned Occlusion Reasoning for Pepper Plants](https://mlsanity.com/paper/2603.06512/) (2026)
- [Detector Confidence Signals Presence Rather Than Occlusion in Cluttered Manipulation](https://mlsanity.com/paper/2607.13361/) (2026)
- [FUSE: Active Functional Affordance Grounding through Adaptive Semantic-Geometric Evidence Acquisition](https://mlsanity.com/paper/2608.12683/) (2026)
- [INSPECT: Learning Robot View Selection from Assistant Use](https://mlsanity.com/paper/2609.20615/) (2026)
- [Beyond the Current Scene: Event-Referential Grasping with Active View Selection](https://mlsanity.com/paper/2609.39375/) (2026)

## Abstract (arXiv)

We propose VISO-Grasp, a novel vision-language-informed system designed to systematically address visibility constraints for grasping in severely occluded environments. By leveraging Foundation Models (FMs) for spatial reasoning and active view planning, our framework constructs and updates an instance-centric representation of spatial relationships, enhancing grasp success under challenging occlusions. Furthermore, this representation facilitates active Next-Best-View (NBV) planning and optimizes sequential grasping strategies when direct grasping is infeasible. Additionally, we introduce a multi-view uncertainty-driven grasp fusion mechanism that refines grasp confidence and directional uncertainty in real-time, ensuring robust and stable grasp execution. Extensive real-world experiments demonstrate that VISO-Grasp achieves a success rate of 87.5% in target-oriented grasping with the fewest grasp attempts outperforming baselines. To the best of our knowledge, VISO-Grasp is the first unified framework integrating FMs into target-aware active view planning and 6-DoF grasping in environments with severe occlusions and entire invisibility constraints. Code is available at: https://github.com/YitianShi/vMF-Contact