Betsu-Betsu: Multi-View Separable 3D Reconstruction of Two Interacting Objects
Not on the map. Why?
Abstract (arXiv)
Separable 3D reconstruction of multiple objects from multi-view RGB images -- resulting in two different 3D shapes for the two objects with a clear separation between them -- remains a sparsely researched problem. It is challenging due to severe mutual occlusions and ambiguities along the objects' interaction boundaries. This paper investigates the setting and introduces a new neuro-implicit method that can reconstruct the geometry and appearance of two objects undergoing close interactions while disjoining both in 3D, avoiding surface inter-penetrations and enabling novel-view synthesis of the observed scene. The framework is end-to-end trainable and supervised using a novel alpha-blending regularisation that ensures that the two geometries are well separated even under extreme occlusions. Our reconstruction method is markerless and can be applied to rigid as well as articulated objects. We introduce a new dataset consisting of close interactions between a human and an object and also evaluate on two scenes of humans performing martial arts. The experiments confirm the effectiveness of our framework and substantial improvements using 3D and novel view synthesis metrics compared to several existing approaches applicable in our setting.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 46 of its 53 references that are in the map. First posted on arXiv 19 Feb 2025.
Builds on
46 of 53 references.
References · 46
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
- 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,712
- 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,612
- 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
- 2021Volume Rendering of Neural Implicit Surfacescited by 996
- 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 646
- 2020Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humanscited by 634
- 2022HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Videocited by 483
- 2020Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Videocited by 471
- 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 415
34 more
- 2020GRAB: A Dataset of Whole-Body Human Grasping of Objectscited by 405
- 2021Advances in Neural Renderingcited by 365
- 2021DexYCB: A Benchmark for Capturing Hand Grasping of Objectscited by 325
- 2022NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstructioncited by 314
- 2022ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulationcited by 312
- 2019FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Imagescited by 271
- 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
- 2022BEHAVE: Dataset and Method for Tracking Human Object Interactionscited by 249
- 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 230
- 2020ContactPose: A Dataset of Grasps with Object Contact and Hand Posecited by 183
- 2020Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wildcited by 152
- 2021Editable Free-viewpoint Video Using a Layered Neural Representationcited by 148
- 2024RGBD Objects in the Wild: Scaling Real-World 3D Object Learning from RGB-D Videoscited by 115
- 2021Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Controlcited by 106
- 2022CHORE: Contact, Human and Object REconstruction from a single RGB imagecited by 95
- 2022InterCap: Joint Markerless 3D Tracking of Humans and Objects in Interactioncited by 93
- 2022What's in your hands? 3D Reconstruction of Generic Objects in Handscited by 90
- 2023Hi4D: 4D Instance Segmentation of Close Human Interactioncited by 87
- 2022Full-Body Articulated Human-Object Interactioncited by 78
- 2022Object-Compositional Neural Implicit Surfacescited by 74
- 2022PermutoSDF: Fast Multi-View Reconstruction with Implicit Surfaces using Permutohedral Latticescited by 73
- 2022NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactionscited by 68
- 2022Human Performance Modeling and Rendering via Neural Animated Meshcited by 64
- 2023ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactionscited by 62
- 2023AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Posecited by 49
- 2022NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactionscited by 44
- 2023ObjectSDF++: Improved Object-Compositional Neural Implicit Surfacescited by 39
- 2021Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactionscited by 37
- 2022Reconstructing Hand-Held Objects from Monocular Videocited by 35
- 2023Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Modelscited by 22
- 2022MoCapDeform: Monocular 3D Human Motion Capture in Deformable Scenescited by 18
- 2023Decaf: Monocular Deformation Capture for Face and Hand Interactionscited by 18
- 2023SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenescited by 11
- 2024MetaCap: Meta-learning Priors from Multi-View Imagery for Sparse-view Human Performance Capture and Renderingcited by 11
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.