# ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulation

arXiv: https://arxiv.org/abs/2203.06856 · HTML: https://mlsanity.com/paper/2203.06856/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 104 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Builds on (most cited first, 30 of 63)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding](https://mlsanity.com/paper/2007.10985/) (2020)
- [Geometric deep learning on graphs and manifolds using mixture model CNNs](https://mlsanity.com/paper/1611.08402/) (2016)
- [OctNet: Learning Deep 3D Representations at High Resolutions](https://mlsanity.com/paper/1611.05009/) (2016)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation](https://mlsanity.com/paper/2007.04954/) (2020)

## Built on by (most cited first, 24 of 24)

- [Causal World Modeling for Robot Control](https://mlsanity.com/paper/2601.21998/) (2026)
- [General Flow as Foundation Affordance for Scalable Robot Learning](https://mlsanity.com/paper/2401.11439/) (2024)
- [Differentiable Physics Simulation of Dynamics-Augmented Neural Objects](https://mlsanity.com/paper/2210.09420/) (2022)
- [EquivAct: SIM(3)-Equivariant Visuomotor Policies beyond Rigid Object Manipulation](https://mlsanity.com/paper/2310.16050/) (2023)
- [Native Video-Action Pretraining for Generalizable Robot Control](https://mlsanity.com/paper/2607.08639/) (2026)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [Learning Foresightful Dense Visual Affordance for Deformable Object Manipulation](https://mlsanity.com/paper/2303.11057/) (2023)
- [GINA-3D: Learning to Generate Implicit Neural Assets in the Wild](https://mlsanity.com/paper/2304.02163/) (2023)
- [DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics](https://mlsanity.com/paper/2304.03223/) (2023)
- [Dynamic-Resolution Model Learning for Object Pile Manipulation](https://mlsanity.com/paper/2306.16700/) (2023)
- [Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulation](https://mlsanity.com/paper/2412.12089/) (2024)
- [NeRFs in Robotics: A Survey](https://mlsanity.com/paper/2405.01333/) (2024)
- [NeuralGrasps: Learning Implicit Representations for Grasps of Multiple Robotic Hands](https://mlsanity.com/paper/2207.02959/) (2022)
- [NeuSE: Neural SE(3)-Equivariant Embedding for Long-Term Object-Based Simultaneous Localization and Mapping](https://mlsanity.com/paper/2303.07308/) (2023)
- [DoughNet: A Visual Predictive Model for Topological Manipulation of Deformable Objects](https://mlsanity.com/paper/2404.12524/) (2024)
- [SculptDiff: Learning Robotic Clay Sculpting from Humans with Goal Conditioned Diffusion Policy](https://mlsanity.com/paper/2403.10401/) (2024)
- [Doduo: Learning Dense Visual Correspondence from Unsupervised Semantic-Aware Flow](https://mlsanity.com/paper/2309.15110/) (2023)
- [Differentiable Physics-based System Identification for Robotic Manipulation of Elastoplastic Materials](https://mlsanity.com/paper/2411.00554/) (2024)
- [Learning to Efficiently Plan Robust Frictional Multi-Object Grasps](https://mlsanity.com/paper/2210.07420/) (2022)
- [DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects](https://mlsanity.com/paper/2305.04449/) (2023)
- [NeuROK: Generative 4D Neural Object Kinematics](https://mlsanity.com/paper/2605.30347/) (2026)
- [Robustness without Wrinkles: Parallel Simulation and Robust MPC for Certified Deformable Manipulation](https://mlsanity.com/paper/2606.14188/) (2026)
- [Implicit Neural-Representation Learning for Elastic Deformable-Object Manipulations](https://mlsanity.com/paper/2505.00500/) (2025)
- [Instance-Agnostic Geometry and Contact Dynamics Learning](https://mlsanity.com/paper/2309.05832/) (2023)

## Abstract (arXiv)

Manipulating volumetric deformable objects in the real world, like plush toys and pizza dough, bring substantial challenges due to infinite shape variations, non-rigid motions, and partial observability. We introduce ACID, an action-conditional visual dynamics model for volumetric deformable objects based on structured implicit neural representations. ACID integrates two new techniques: implicit representations for action-conditional dynamics and geodesics-based contrastive learning. To represent deformable dynamics from partial RGB-D observations, we learn implicit representations of occupancy and flow-based forward dynamics. To accurately identify state change under large non-rigid deformations, we learn a correspondence embedding field through a novel geodesics-based contrastive loss. To evaluate our approach, we develop a simulation framework for manipulating complex deformable shapes in realistic scenes and a benchmark containing over 17,000 action trajectories with six types of plush toys and 78 variants. Our model achieves the best performance in geometry, correspondence, and dynamics predictions over existing approaches. The ACID dynamics models are successfully employed to goal-conditioned deformable manipulation tasks, resulting in a 30% increase in task success rate over the strongest baseline. Furthermore, we apply the simulation-trained ACID model directly to real-world objects and show success in manipulating them into target configurations. For more results and information, please visit https://b0ku1.github.io/acid/ .