# Fast Point Cloud to Mesh Reconstruction for Deformable Object Tracking arXiv: https://arxiv.org/abs/2311.02749 ยท HTML: https://mlsanity.com/paper/2311.02749/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 21 of its 31 references that are in the map. - First posted on arXiv 5 Nov 2023. ## Builds on (most cited first, 21 of 21) - [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018) - [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016) - [PCN: Point Completion Network](https://mlsanity.com/paper/1808.00671/) (2018) - [Shape As Points: A Differentiable Poisson Solver](https://mlsanity.com/paper/2106.03452/) (2021) - [GRNet: Gridding Residual Network for Dense Point Cloud Completion](https://mlsanity.com/paper/2006.03761/) (2020) - [4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface](https://mlsanity.com/paper/2105.01905/) (2021) - [Shape2Motion: Joint Analysis of Motion Parts and Attributes from 3D Shapes](https://mlsanity.com/paper/1903.03911/) (2019) - [Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks](https://mlsanity.com/paper/2103.10429/) (2021) - [Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols](https://mlsanity.com/paper/1502.03143/) (2015) - [Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction](https://mlsanity.com/paper/2012.01451/) (2020) - [Occupancy Flow Fields for Motion Forecasting in Autonomous Driving](https://mlsanity.com/paper/2203.03875/) (2022) - [Neural Mesh Flow: 3D Manifold Mesh Generation via Diffeomorphic Flows](https://mlsanity.com/paper/2007.10973/) (2020) - [3DN: 3D Deformation Network](https://mlsanity.com/paper/1903.03322/) (2019) - [CaDeX: Learning Canonical Deformation Coordinate Space for Dynamic Surface Representation via Neural Homeomorphism](https://mlsanity.com/paper/2203.16529/) (2022) - [Point2Mesh: A Self-Prior for Deformable Meshes](https://mlsanity.com/paper/2005.11084/) (2020) - [Topology-Preserving Shape Reconstruction and Registration via Neural Diffeomorphic Flow](https://mlsanity.com/paper/2203.08652/) (2022) - [Learning Parallel Dense Correspondence from Spatio-Temporal Descriptors for Efficient and Robust 4D Reconstruction](https://mlsanity.com/paper/2103.16341/) (2021) - [Deformation-Aware 3D Model Embedding and Retrieval](https://mlsanity.com/paper/2004.01228/) (2020) - [ShapeFlow: Learnable Deformations Among 3D Shapes](https://mlsanity.com/paper/2006.07982/) (2020) - [Learning Compositional Representation for 4D Captures with Neural ODE](https://mlsanity.com/paper/2103.08271/) (2021) - [RFNet-4D++: Joint Object Reconstruction and Flow Estimation from 4D Point Clouds with Cross-Attention Spatio-Temporal Features](https://mlsanity.com/paper/2203.16482/) (2022) ## Built on by (most cited first, 5 of 5) - [PokeFlex: A Real-World Dataset of Volumetric Deformable Objects for Robotics](https://mlsanity.com/paper/2410.07688/) (2024) - [RaggeDi: Diffusion-based State Estimation of Disordered Rags, Sheets, Towels and Blankets](https://mlsanity.com/paper/2409.11831/) (2024) - [Point Cloud to Mesh Reconstruction: Methods, Trade-offs, and Implementation Guide](https://mlsanity.com/paper/2412.10977/) (2024) - [Point Cloud Sequence Encoding for Material-conditioned Graph Network Simulators](https://mlsanity.com/paper/2605.20978/) (2026) - [Cepheid: Decoupling Scale from Structure for Cross-Family Motion Completion](https://mlsanity.com/paper/2610.18871/) (2026) ## Abstract (arXiv) The world around us is full of soft objects we perceive and deform with dexterous hand movements. For a robotic hand to control soft objects, it has to acquire online state feedback of the deforming object. While RGB-D cameras can collect occluded point clouds at a rate of 30Hz, this does not represent a continuously trackable object surface. Hence, in this work, we developed a method that takes as input a template mesh which is the mesh of an object in its non-deformed state and a deformed point cloud of the same object, and then shapes the template mesh such that it matches the deformed point cloud. The reconstruction of meshes from point clouds has long been studied in the field of Computer graphics under 3D reconstruction and 4D reconstruction, however, both lack the speed and generalizability needed for robotics applications. Our model is designed using a point cloud auto-encoder and a Real-NVP architecture. Our trained model can perform mesh reconstruction and tracking at a rate of 58Hz on a template mesh of 3000 vertices and a deformed point cloud of 5000 points and is generalizable to the deformations of six different object categories which are assumed to be made of soft material in our experiments (scissors, hammer, foam brick, cleanser bottle, orange, and dice). The object meshes are taken from the YCB benchmark dataset. An instance of a downstream application can be the control algorithm for a robotic hand that requires online feedback from the state of the manipulated object which would allow online grasp adaptation in a closed-loop manner. Furthermore, the tracking capacity of our method can help in the system identification of deforming objects in a marker-free approach. In future work, we will extend our trained model to generalize beyond six object categories and additionally to real-world deforming point clouds.