# GraspNeRF: Multiview-based 6-DoF Grasp Detection for Transparent and Specular Objects Using Generalizable NeRF

arXiv: https://arxiv.org/abs/2210.06575 · HTML: https://mlsanity.com/paper/2210.06575/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 17 of 17)

- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes](https://mlsanity.com/paper/2103.14127/) (2021)
- [Neural Rays for Occlusion-aware Image-based Rendering](https://mlsanity.com/paper/2107.13421/) (2021)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects](https://mlsanity.com/paper/2110.14217/) (2021)
- [Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations](https://mlsanity.com/paper/2104.01542/) (2021)
- [Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter](https://mlsanity.com/paper/2101.01132/) (2021)
- [NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields](https://mlsanity.com/paper/2203.01913/) (2022)
- [TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and a Grasping Baseline](https://mlsanity.com/paper/2202.08471/) (2022)
- [Domain Randomization-Enhanced Depth Simulation and Restoration for Perceiving and Grasping Specular and Transparent Objects](https://mlsanity.com/paper/2208.03792/) (2022)
- [RGB Matters: Learning 7-DoF Grasp Poses on Monocular RGBD Images](https://mlsanity.com/paper/2103.02184/) (2021)
- [Multi-modal Transfer Learning for Grasping Transparent and Specular Objects](https://mlsanity.com/paper/2006.00028/) (2020)
- [GlassLoc: Plenoptic Grasp Pose Detection in Transparent Clutter](https://mlsanity.com/paper/1909.04269/) (2019)
- [Fuzzy-Depth Objects Grasping Based on FSG Algorithm and a Soft Robotic Hand](https://mlsanity.com/paper/2110.10924/) (2021)

## Built on by (most cited first, 30 of 67)

- [Text-to-3D using Gaussian Splatting](https://mlsanity.com/paper/2309.16585/) (2023)
- [Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation](https://mlsanity.com/paper/2308.07931/) (2023)
- [UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learning](https://mlsanity.com/paper/2304.00464/) (2023)
- [Renderable Neural Radiance Map for Visual Navigation](https://mlsanity.com/paper/2303.00304/) (2023)
- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [RealDex: Towards Human-like Grasping for Robotic Dexterous Hand](https://mlsanity.com/paper/2402.13853/) (2024)
- [FP3: A 3D Foundation Policy for Robotic Manipulation](https://mlsanity.com/paper/2503.08950/) (2025)
- [What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Models](https://mlsanity.com/paper/2512.03422/) (2025)
- [Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Survey](https://mlsanity.com/paper/2507.14501/) (2025)
- [UFORecon: Generalizable Sparse-View Surface Reconstruction from Arbitrary and UnFavOrable Sets](https://mlsanity.com/paper/2403.05086/) (2024)
- [An Economic Framework for 6-DoF Grasp Detection](https://mlsanity.com/paper/2407.08366/) (2024)
- [GAMMA: Graspability-Aware Mobile MAnipulation Policy Learning based on Online Grasping Pose Fusion](https://mlsanity.com/paper/2309.15459/) (2023)
- [SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulation](https://mlsanity.com/paper/2310.16838/) (2023)
- [Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding](https://mlsanity.com/paper/2408.13024/) (2024)
- [Neural Fields in Robotics: A Survey](https://mlsanity.com/paper/2410.20220/) (2024)
- [NeRFs in Robotics: A Survey](https://mlsanity.com/paper/2405.01333/) (2024)
- [Open-vocabulary Mobile Manipulation in Unseen Dynamic Environments with 3D Semantic Maps](https://mlsanity.com/paper/2406.18115/) (2024)
- [Neural Radiance Fields for the Real World: A Survey](https://mlsanity.com/paper/2501.13104/) (2025)
- [Generalizing 6-DoF Grasp Detection via Domain Prior Knowledge](https://mlsanity.com/paper/2404.01727/) (2024)
- [VISO-Grasp: Vision-Language Informed Spatial Object-centric 6-DoF Active View Planning and Grasping in Clutter and Invisibility](https://mlsanity.com/paper/2503.12609/) (2025)
- [ASGrasp: Generalizable Transparent Object Reconstruction and 6-DoF Grasp Detection from RGB-D Active Stereo Camera](https://mlsanity.com/paper/2405.05648/) (2024)
- [Lightweight Language-driven Grasp Detection using Conditional Consistency Model](https://mlsanity.com/paper/2407.17967/) (2024)
- [ClearDepth: Enhanced Stereo Perception of Transparent Objects for Robotic Manipulation](https://mlsanity.com/paper/2409.08926/) (2024)
- [ScrewSplat: An End-to-End Method for Articulated Object Recognition](https://mlsanity.com/paper/2508.02146/) (2025)
- [BAA-NGP: Bundle-Adjusting Accelerated Neural Graphics Primitives](https://mlsanity.com/paper/2306.04166/) (2023)
- [Turbo-GS: Accelerating 3D Gaussian Fitting for High-Quality Radiance Fields](https://mlsanity.com/paper/2412.13547/) (2024)
- [Monocular Depth Estimation and Segmentation for Transparent Object with Iterative Semantic and Geometric Fusion](https://mlsanity.com/paper/2502.14616/) (2025)
- [ImageManip: Image-based Robotic Manipulation with Affordance-guided Next View Selection](https://mlsanity.com/paper/2310.09069/) (2023)
- [TransDiff: Diffusion-Based Method for Manipulating Transparent Objects Using a Single RGB-D Image](https://mlsanity.com/paper/2503.12779/) (2025)
- [RFTrans: Leveraging Refractive Flow of Transparent Objects for Surface Normal Estimation and Manipulation](https://mlsanity.com/paper/2311.12398/) (2023)

## Abstract (arXiv)

In this work, we tackle 6-DoF grasp detection for transparent and specular objects, which is an important yet challenging problem in vision-based robotic systems, due to the failure of depth cameras in sensing their geometry. We, for the first time, propose a multiview RGB-based 6-DoF grasp detection network, GraspNeRF, that leverages the generalizable neural radiance field (NeRF) to achieve material-agnostic object grasping in clutter. Compared to the existing NeRF-based 3-DoF grasp detection methods that rely on densely captured input images and time-consuming per-scene optimization, our system can perform zero-shot NeRF construction with sparse RGB inputs and reliably detect 6-DoF grasps, both in real-time. The proposed framework jointly learns generalizable NeRF and grasp detection in an end-to-end manner, optimizing the scene representation construction for the grasping. For training data, we generate a large-scale photorealistic domain-randomized synthetic dataset of grasping in cluttered tabletop scenes that enables direct transfer to the real world. Our extensive experiments in synthetic and real-world environments demonstrate that our method significantly outperforms all the baselines in all the experiments while remaining in real-time. Project page can be found at https://pku-epic.github.io/GraspNeRF