mlsanityData to 7 Oct 2026

3D Reconstruction of Objects in Hands without Real World 3D Supervision

arXiv 2305.03036 ↗First posted 4 May 2023cs.CV cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Prior works for reconstructing hand-held objects from a single image train models on images paired with 3D shapes. Such data is challenging to gather in the real world at scale. Consequently, these approaches do not generalize well when presented with novel objects in in-the-wild settings. While 3D supervision is a major bottleneck, there is an abundance of a) in-the-wild raw video data showing hand-object interactions and b) synthetic 3D shape collections. In this paper, we propose modules to leverage 3D supervision from these sources to scale up the learning of models for reconstructing hand-held objects. Specifically, we extract multiview 2D mask supervision from videos and 3D shape priors from shape collections. We use these indirect 3D cues to train occupancy networks that predict the 3D shape of objects from a single RGB image. Our experiments in the challenging object generalization setting on in-the-wild MOW dataset show 11.6% relative improvement over models trained with 3D supervision on existing datasets.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 59 of its 83 references that are in the map. First posted on arXiv 4 May 2023.

Builds on

59 of 83 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  3. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  4. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  5. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  7. 2016Least Squares Generative Adversarial Networkscited by 1,000
  8. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  9. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  10. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
  11. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  12. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
38 more
  1. 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 397
  2. 2019Learning joint reconstruction of hands and manipulated objectscited by 377
  3. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  4. 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 320
  5. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  6. 2017CORe50: a New Dataset and Benchmark for Continuous Object Recognitioncited by 287
  7. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  8. 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
  9. 2019What Do Single-view 3D Reconstruction Networks Learn?cited by 259
  10. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  11. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  12. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  13. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  14. 2022SPARF: Neural Radiance Fields from Sparse and Noisy Posescited by 195
  15. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  16. 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 180
  17. 2020Grasping Field: Learning Implicit Representations for Human Graspscited by 178
  18. 2021Hand-Object Contact Consistency Reasoning for Human Grasps Generationcited by 160
  19. 2019ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imagingcited by 145
  20. 2020Reconstructing Hand-Object Interactions in the Wildcited by 143
  21. 2020Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstructioncited by 139
  22. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 139
  23. 2022OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interactioncited by 129
  24. 2021Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimationcited by 122
  25. 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
  26. 2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 83
  27. 2020Learning Dexterous Grasping with Object-Centric Visual Affordancescited by 80
  28. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 79
  29. 2020FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integrationcited by 72
  30. 2022ShAPO: Implicit Representations for Multi-Object Shape, Appearance, and Pose Optimizationcited by 70
  31. 2021Shelf-Supervised Mesh Prediction in the Wildcited by 65
  32. 2023CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objectscited by 64
  33. 2022AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstructioncited by 58
  34. 2014Category-Specific Object Reconstruction from a Single Imagecited by 56
  35. 2021NeuralDiff: Segmenting 3D objects that move in egocentric videoscited by 56
  36. 2018Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Predictioncited by 52
  37. 2022Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Handscited by 49
  38. 2023gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstructioncited by 49

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 5; 2026: 4.

12 papers.

  1. 2023HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Videocited by 76
  2. 2024Reconstructing Hand-Held Objects in 3D from Images and Videoscited by 23
  3. 2023Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videoscited by 21
  4. 2025HORT: Monocular Hand-held Objects Reconstruction with Transformerscited by 11
  5. 2025How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactionscited by 10
  6. 2026Do as I Do: Dexterous Manipulation Data from Everyday Human Videoscited by 9
  7. 2026ForeHOI: Feed-forward 3D Object Reconstruction from Daily Hand-Object Interaction Videoscited by 6
  8. 2025Towards Dynamic 3D Reconstruction of Hand-Instrument Interaction in Ophthalmic Surgerycited by 2
  9. 2025Reconstructing Objects along Hand Interaction Timelines in Egocentric Videocited by 2
  10. 2025VPHO: Joint Visual-Physical Cue Learning and Aggregation for Hand-Object Pose Estimationcited by 0
  11. 2026Towards in-the-wild Egocentric 3D Hand-Object Pose Estimationcited by 0
  12. 2026DexAgent: An Agentic Human2Sim2Robot Framework for Dexterous Manipulation with Self-Evolving Tool Librarycited by 0