# DeformNet: Latent Space Modeling and Dynamics Prediction for Deformable Object Manipulation

arXiv: https://arxiv.org/abs/2402.07648 · HTML: https://mlsanity.com/paper/2402.07648/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2024.

## Built on by (most cited first, 10 of 10)

- [Causal World Modeling for Robot Control](https://mlsanity.com/paper/2601.21998/) (2026)
- [Native Video-Action Pretraining for Generalizable Robot Control](https://mlsanity.com/paper/2607.08639/) (2026)
- [Cloning Deterministic Worlds: The Critical Role of Latent Geometry in Long-Horizon World Models](https://mlsanity.com/paper/2510.26782/) (2025)
- [NeuROK: Generative 4D Neural Object Kinematics](https://mlsanity.com/paper/2605.30347/) (2026)
- [Robustness without Wrinkles: Parallel Simulation and Robust MPC for Certified Deformable Manipulation](https://mlsanity.com/paper/2606.14188/) (2026)
- [Implicit Neural-Representation Learning for Elastic Deformable-Object Manipulations](https://mlsanity.com/paper/2505.00500/) (2025)
- [Manipulating Elasto-Plastic Objects With 3D Occupancy and Learning-Based Predictive Control](https://mlsanity.com/paper/2505.16249/) (2025)
- [CLASP: General-Purpose Clothes Manipulation with Semantic Keypoints](https://mlsanity.com/paper/2507.19983/) (2025)
- [DexKnot: Generalizable Visuomotor Policy Learning for Dexterous Bag-Knotting Manipulation](https://mlsanity.com/paper/2603.07136/) (2026)
- [Rendering Multi-Human and Multi-Object with 3D Gaussian Splatting](https://mlsanity.com/paper/2604.02996/) (2026)

## Abstract (arXiv)

Manipulating deformable objects is a ubiquitous task in household environments, demanding adequate representation and accurate dynamics prediction due to the objects' infinite degrees of freedom. This work proposes DeformNet, which utilizes latent space modeling with a learned 3D representation model to tackle these challenges effectively. The proposed representation model combines a PointNet encoder and a conditional neural radiance field (NeRF), facilitating a thorough acquisition of object deformations and variations in lighting conditions. To model the complex dynamics, we employ a recurrent state-space model (RSSM) that accurately predicts the transformation of the latent representation over time. Extensive simulation experiments with diverse objectives demonstrate the generalization capabilities of DeformNet for various deformable object manipulation tasks, even in the presence of previously unseen goals. Finally, we deploy DeformNet on an actual UR5 robotic arm to demonstrate its capability in real-world scenarios.