mlsanityData to 7 Oct 2026

RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control

arXiv 2210.11668 ↗First posted 21 Oct 2022cs.RO cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

We present a system for collision-free control of a robot manipulator that uses only RGB views of the world. Perceptual input of a tabletop scene is provided by multiple images of an RGB camera (without depth) that is either handheld or mounted on the robot end effector. A NeRF-like process is used to reconstruct the 3D geometry of the scene, from which the Euclidean full signed distance function (ESDF) is computed. A model predictive control algorithm is then used to control the manipulator to reach a desired pose while avoiding obstacles in the ESDF. We show results on a real dataset collected and annotated in our lab.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 39 of its 57 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

39 of 57 references.

References · 39

  1. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  2. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  3. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  4. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  5. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  6. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  7. 2022Point-NeRF: Point-based Neural Radiance Fieldscited by 503
  8. 2022MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstructioncited by 467
  9. 2021Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapescited by 426
  10. 2021Neural RGB-D Surface Reconstructioncited by 312
  11. 2020CosyPose: Consistent multi-view multi-object 6D pose estimationcited by 287
  12. 2020MetaSDF: Meta-learning Signed Distance Functionscited by 214
27 more
  1. 2021Vision-Only Robot Navigation in a Neural Radiance Worldcited by 153
  2. 2018CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAMcited by 151
  3. 2016Voxblox: Incremental 3D Euclidean Signed Distance Fields for On-Board MAV Planningcited by 148
  4. 2019ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulationcited by 124
  5. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  6. 2022iSDF: Real-Time Neural Signed Distance Fields for Robot Perceptioncited by 107
  7. 20226-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmarkcited by 85
  8. 2022You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstrationcited by 79
  9. 2021CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulationcited by 49
  10. 2020MoreFusion: Multi-object Reasoning for 6D Pose Estimation from Volumetric Fusioncited by 46
  11. 2021STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulationcited by 46
  12. 2018Riemannian Motion Policiescited by 45
  13. 2020Object Rearrangement Using Learned Implicit Collision Functionscited by 43
  14. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  15. 2021Learning Models as Functionals of Signed-Distance Fields for Manipulation Planningcited by 29
  16. 2021Vision-driven Compliant Manipulation for Reliable, High-Precision Assembly Taskscited by 18
  17. 2021Multi-View Fusion for Multi-Level Robotic Scene Understandingcited by 17
  18. 2019Self-Supervised Deep Depth Denoisingcited by 15
  19. 2020Task-driven Perception and Manipulation for Constrained Placement of Unknown Objectscited by 11
  20. 2022Points2NeRF: Generating Neural Radiance Fields from 3D point cloudcited by 11
  21. 2020Voxel Map for Visual SLAMcited by 9
  22. 2022Model Predictive Control for Fluid Human-to-Robot Handoverscited by 9
  23. 2022Sampling-free obstacle gradients and reactive planning in Neural Radiance Fields (NeRF)cited by 6
  24. 2021DiffCo: Auto-Differentiable Proxy Collision Detection with Multi-class Labels for Safety-Aware Trajectory Optimizationcited by 4
  25. 2016An Approach for Noise Removal on Depth Imagescited by 2
  26. 2022One Object at a Time: Accurate and Robust Structure From Motion for Robotscited by 2
  27. 2020Unsupervised Enhancement of Real-World Depth Images Using Tri-Cycle GANcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 3; 2025: 3.

10 papers.

  1. 2023HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose Annotations, Affordances, and Reconstructionscited by 47
  2. 2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
  3. 2024NeRFs in Robotics: A Surveycited by 16
  4. 2025IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environmentscited by 11
  5. 2023HandNeRF: Learning to Reconstruct Hand-Object Interaction Scene from a Single RGB Imagecited by 10
  6. 2023BAA-NGP: Bundle-Adjusting Accelerated Neural Graphics Primitivescited by 8
  7. 2025RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completioncited by 5
  8. 2025TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulationcited by 2
  9. 2024NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flowscited by 1
  10. 2024GUMBEL-NERF: Representing Unseen Objects as Part-Compositional Neural Radiance Fieldscited by 0