mlsanityData to 7 Oct 2026

Perceiving Unseen 3D Objects by Poking the Objects

arXiv 2302.13375 ↗First posted 26 Feb 2023cs.RO cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

We present a novel approach to interactive 3D object perception for robots. Unlike previous perception algorithms that rely on known object models or a large amount of annotated training data, we propose a poking-based approach that automatically discovers and reconstructs 3D objects. The poking process not only enables the robot to discover unseen 3D objects but also produces multi-view observations for 3D reconstruction of the objects. The reconstructed objects are then memorized by neural networks with regular supervised learning and can be recognized in new test images. The experiments on real-world data show that our approach could unsupervisedly discover and reconstruct unseen 3D objects with high quality, and facilitate real-world applications such as robotic grasping. The code and supplementary materials are available at the project page: https://zju3dv.github.io/poking_perception.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 40 of its 53 references that are in the map. First posted on arXiv 26 Feb 2023.

Builds on

40 of 53 references.

References · 40

  1. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  2. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  3. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  5. 2015Fast R-CNNcited by 2,930
  6. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  7. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  8. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  9. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  10. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  11. 2020Implicit Geometric Regularization for Learning Shapescited by 707
  12. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 641
28 more
  1. 2019Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimationcited by 490
  2. 2018PVNet: Pixel-wise Voting Network for 6DoF Pose Estimationcited by 385
  3. 2020Neural Scene Graphs for Dynamic Scenescited by 298
  4. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  5. 2021Conditional Object-Centric Learning from Videocited by 247
  6. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  7. 2021NeRF--: Neural Radiance Fields Without Known Camera Parameterscited by 155
  8. 2022OnePose: One-Shot Object Pose Estimation without CAD Modelscited by 155
  9. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 152
  10. 2018Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approachcited by 117
  11. 2023OnePose++: Keypoint-Free One-Shot Object Pose Estimation without CAD Modelscited by 114
  12. 2018MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objectscited by 95
  13. 2022OSOP: A Multi-Stage One Shot Object Pose Estimation Frameworkcited by 81
  14. 2018PointNetGPD: Detecting Grasp Configurations from Point Setscited by 80
  15. 2016Interactive Perception: Leveraging Action in Perception and Perception in Actioncited by 77
  16. 2020STaR: Self-supervised Tracking and Reconstruction of Rigid Objects in Motion with Neural Renderingcited by 76
  17. 2018MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAMcited by 65
  18. 2019DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactionscited by 61
  19. 2018Reasoning About Physical Interactions with Object-Oriented Prediction and Planningcited by 51
  20. 2019S4G: Amodal Single-view Single-Shot SE(3) Grasp Detection in Cluttered Scenescited by 51
  21. 2020FroDO: From Detections to 3D Objectscited by 51
  22. 2020Disp R-CNN: Stereo 3D Object Detection via Shape Prior Guided Instance Disparity Estimationcited by 43
  23. 2020Learning 3D Dynamic Scene Representations for Robot Manipulationcited by 39
  24. 2019Learning Continuous 3D Reconstructions for Geometrically Aware Graspingcited by 28
  25. 2018Fully Convolutional Grasp Detection Network with Oriented Anchor Boxcited by 27
  26. 2020Unsupervised Discovery of 3D Physical Objects from Videocited by 27
  27. 2019Object-centric Forward Modeling for Model Predictive Controlcited by 19
  28. 2020DGCM-Net: Dense Geometrical Correspondence Matching Network for Incremental Experience-based Robotic Graspingcited by 11

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 4.

5 papers.

  1. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  2. 2023EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Explorationcited by 16
  3. 2024NeRFs in Robotics: A Surveycited by 16
  4. 2024Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Reviewcited by 9
  5. 2024NeRF-Aug: Data Augmentation for Robotics with Neural Radiance Fieldscited by 1