# LUDO: Low-Latency Understanding of Deformable Objects using Point Cloud Occupancy Functions

arXiv: https://arxiv.org/abs/2411.08777 · HTML: https://mlsanity.com/paper/2411.08777/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 13 Nov 2024.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Visualizing Deep Neural Network Decisions: Prediction Difference Analysis](https://mlsanity.com/paper/1702.04595/) (2017)
- [Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation](https://mlsanity.com/paper/1806.08756/) (2018)
- [A Survey of Non-Rigid 3D Registration](https://mlsanity.com/paper/2203.07858/) (2022)
- [Movement Primitive Diffusion: Learning Gentle Robotic Manipulation of Deformable Objects](https://mlsanity.com/paper/2312.10008/) (2023)
- [Sim-To-Real Transfer for Visual Reinforcement Learning of Deformable Object Manipulation for Robot-Assisted Surgery](https://mlsanity.com/paper/2406.06092/) (2024)
- [Learning Soft Tissue Behavior of Organs for Surgical Navigation with Convolutional Neural Networks](https://mlsanity.com/paper/1904.00722/) (2019)
- [Non-Rigid Volume to Surface Registration using a Data-Driven Biomechanical Model](https://mlsanity.com/paper/2005.14695/) (2020)
- [Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds](https://mlsanity.com/paper/2110.04685/) (2021)
- [Sim-to-Real Surgical Robot Learning and Autonomous Planning for Internal Tissue Points Manipulation using Reinforcement Learning](https://mlsanity.com/paper/2306.14085/) (2023)
- [Registered and Segmented Deformable Object Reconstruction from a Single View Point Cloud](https://mlsanity.com/paper/2311.07357/) (2023)
- [LOOC: Localizing Organs using Occupancy Networks and Body Surface Depth Images](https://mlsanity.com/paper/2406.12407/) (2024)
- [Tracking Tumors under Deformation from Partial Point Clouds using Occupancy Networks](https://mlsanity.com/paper/2411.02619/) (2024)
- [Multi-tap Resistive Sensing and FEM Modeling enables Shape and Force Estimation in Soft Robots](https://mlsanity.com/paper/2311.14566/) (2023)

## Built on by (most cited first, 4 of 4)

- [A Survey of Medical Point Cloud Shape Learning: Registration, Reconstruction and Variation](https://mlsanity.com/paper/2508.03057/) (2025)
- [Deep learning based Non-Rigid Volume-to-Surface Registration for Brain Shift compensation Using Point Cloud](https://mlsanity.com/paper/2604.17389/) (2026)
- [Comparing Commercial Depth Sensor Accuracy for Medical Applications](https://mlsanity.com/paper/2606.13028/) (2026)
- [Occupancy Network-Guided Autonomous Robotic Partial Nephrectomy](https://mlsanity.com/paper/2609.16186/) (2026)

## Abstract (arXiv)

Accurately determining the shape of deformable objects and the location of their internal structures is crucial for medical tasks that require precise targeting, such as robotic biopsies. We introduce LUDO, a method for accurate low-latency understanding of deformable objects. LUDO reconstructs objects in their deformed state, including their internal structures, from a single-view point cloud observation in under 30 ms using occupancy networks. LUDO provides uncertainty estimates for its predictions. Additionally, it provides explainability by highlighting key features in its input observations. Both uncertainty and explainability are important for safety-critical applications such as surgery. We evaluate LUDO in real-world robotic experiments, achieving a success rate of 98.9% for puncturing various regions of interest (ROIs) inside deformable objects. We compare LUDO to a popular baseline and show its superior ROI localization accuracy, training time, and memory requirements. LUDO demonstrates the potential to interact with deformable objects without the need for deformable registration methods.