# DoughNet: A Visual Predictive Model for Topological Manipulation of Deformable Objects

arXiv: https://arxiv.org/abs/2404.12524 · HTML: https://mlsanity.com/paper/2404.12524/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 18 Apr 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models](https://mlsanity.com/paper/2301.11445/) (2023)
- [Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation](https://mlsanity.com/paper/2306.17115/) (2023)
- [Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids](https://mlsanity.com/paper/1810.01566/) (2018)
- [Masked World Models for Visual Control](https://mlsanity.com/paper/2206.14244/) (2022)
- [PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification](https://mlsanity.com/paper/2303.05512/) (2023)
- [Generalization in Generation: A closer look at Exposure Bias](https://mlsanity.com/paper/1910.00292/) (2019)
- [RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Tools](https://mlsanity.com/paper/2306.14447/) (2023)
- [RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networks](https://mlsanity.com/paper/2205.02909/) (2022)
- [DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting](https://mlsanity.com/paper/2105.12244/) (2021)
- [Learning Multi-Object Dynamics with Compositional Neural Radiance Fields](https://mlsanity.com/paper/2202.11855/) (2022)
- [Visual Grounding of Learned Physical Models](https://mlsanity.com/paper/2004.13664/) (2020)
- [Differentiable Physics Simulation of Dynamics-Augmented Neural Objects](https://mlsanity.com/paper/2210.09420/) (2022)
- [DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools](https://mlsanity.com/paper/2203.17275/) (2022)
- [VIRDO: Visio-tactile Implicit Representations of Deformable Objects](https://mlsanity.com/paper/2202.00868/) (2022)
- [ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation](https://mlsanity.com/paper/2203.06856/) (2022)
- [DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics](https://mlsanity.com/paper/2304.03223/) (2023)
- [VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects](https://mlsanity.com/paper/2210.03701/) (2022)
- [Learning Closed-loop Dough Manipulation Using a Differentiable Reset Module](https://mlsanity.com/paper/2207.04638/) (2022)
- [SculptBot: Pre-Trained Models for 3D Deformable Object Manipulation](https://mlsanity.com/paper/2309.08728/) (2023)
- [Deformable Elasto-Plastic Object Shaping using an Elastic Hand and Model-Based Reinforcement Learning](https://mlsanity.com/paper/2107.06924/) (2021)
- [Learning the Dynamics of Compliant Tool-Environment Interaction for Visuo-Tactile Contact Servoing](https://mlsanity.com/paper/2210.03836/) (2022)

## Built on by (most cited first, 9 of 9)

- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videos](https://mlsanity.com/paper/2506.15680/) (2025)
- [Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulation](https://mlsanity.com/paper/2510.07975/) (2025)
- [Apple: Toward General Active Perception via Reinforcement Learning](https://mlsanity.com/paper/2505.06182/) (2025)
- [Learning Physics-Guided Residual Dynamics for Deformable Object Simulation](https://mlsanity.com/paper/2607.13451/) (2026)
- [PinchBot: Long-Horizon Deformable Manipulation with Guided Diffusion Policy](https://mlsanity.com/paper/2507.17846/) (2025)
- [PhysCoRe: Physics-Corrected Residual World Models for Material-Aware Deformable Dynamics](https://mlsanity.com/paper/2607.20653/) (2026)
- [DexKnot: Generalizable Visuomotor Policy Learning for Dexterous Bag-Knotting Manipulation](https://mlsanity.com/paper/2603.07136/) (2026)
- [Explicit Kinematic Guidance from Analytic Concepts for Vision-Language-Action Models](https://mlsanity.com/paper/2607.26513/) (2026)

## Abstract (arXiv)

Manipulation of elastoplastic objects like dough often involves topological changes such as splitting and merging. The ability to accurately predict these topological changes that a specific action might incur is critical for planning interactions with elastoplastic objects. We present DoughNet, a Transformer-based architecture for handling these challenges, consisting of two components. First, a denoising autoencoder represents deformable objects of varying topology as sets of latent codes. Second, a visual predictive model performs autoregressive set prediction to determine long-horizon geometrical deformation and topological changes purely in latent space. Given a partial initial state and desired manipulation trajectories, it infers all resulting object geometries and topologies at each step. DoughNet thereby allows to plan robotic manipulation; selecting a suited tool, its pose and opening width to recreate robot- or human-made goals. Our experiments in simulated and real environments show that DoughNet is able to significantly outperform related approaches that consider deformation only as geometrical change.