# Ditto: Building Digital Twins of Articulated Objects from Interaction

arXiv: https://arxiv.org/abs/2202.08227 · HTML: https://mlsanity.com/paper/2202.08227/

## Key facts

- Cited by 156 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 63 references that are in the map.
- First posted on arXiv 16 Feb 2022.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Convolutional Occupancy Networks](https://mlsanity.com/paper/2003.04618/) (2020)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [Learning to Estimate 3D Human Pose and Shape from a Single Color Image](https://mlsanity.com/paper/1805.04092/) (2018)
- [BANMo: Building Animatable 3D Neural Models from Many Casual Videos](https://mlsanity.com/paper/2112.12761/) (2021)
- [Neural Articulated Radiance Field](https://mlsanity.com/paper/2104.03110/) (2021)
- [iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks](https://mlsanity.com/paper/2108.03272/) (2021)
- [RoboTHOR: An Open Simulation-to-Real Embodied AI Platform](https://mlsanity.com/paper/2004.06799/) (2020)
- [Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimation](https://mlsanity.com/paper/1808.05942/) (2018)
- [Category-Level Articulated Object Pose Estimation](https://mlsanity.com/paper/1912.11913/) (2019)
- [NASA: Neural Articulated Shape Approximation](https://mlsanity.com/paper/1912.03207/) (2019)

## Built on by (most cited first, 30 of 156)

- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation](https://mlsanity.com/paper/2403.03949/) (2024)
- [PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objects](https://mlsanity.com/paper/2308.07391/) (2023)
- [DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning](https://mlsanity.com/paper/2410.24185/) (2024)
- [URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images](https://mlsanity.com/paper/2405.11656/) (2024)
- [iSDF: Real-Time Neural Signed Distance Fields for Robot Perception](https://mlsanity.com/paper/2204.02296/) (2022)
- [Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Model](https://mlsanity.com/paper/2410.13882/) (2024)
- [Full-Body Articulated Human-Object Interaction](https://mlsanity.com/paper/2212.10621/) (2022)
- [Real2Code: Reconstruct Articulated Objects via Code Generation](https://mlsanity.com/paper/2406.08474/) (2024)
- [SINGAPO: Single Image Controlled Generation of Articulated Parts in Objects](https://mlsanity.com/paper/2410.16499/) (2024)
- [ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splatting](https://mlsanity.com/paper/2502.19459/) (2025)
- [DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects](https://mlsanity.com/paper/2305.05706/) (2023)
- [CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objects](https://mlsanity.com/paper/2303.15782/) (2023)
- [CAGE: Controllable Articulation GEneration](https://mlsanity.com/paper/2312.09570/) (2023)
- [SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fields](https://mlsanity.com/paper/2211.09786/) (2022)
- [RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation](https://mlsanity.com/paper/2402.15487/) (2024)
- [Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/2503.24361/) (2025)
- [REACTO: Reconstructing Articulated Objects from a Single Video](https://mlsanity.com/paper/2404.11151/) (2024)
- [Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modeling](https://mlsanity.com/paper/2502.02590/) (2025)
- [Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perception](https://mlsanity.com/paper/2302.01295/) (2023)
- [MeshArt: Generating Articulated Meshes with Structure-Guided Transformers](https://mlsanity.com/paper/2412.11596/) (2024)
- [OPDMulti: Openable Part Detection for Multiple Objects](https://mlsanity.com/paper/2303.14087/) (2023)
- [Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated Objects](https://mlsanity.com/paper/2309.07473/) (2023)
- [ArticulatedGS: Self-supervised Digital Twin Modeling of Articulated Objects using 3D Gaussian Splatting](https://mlsanity.com/paper/2503.08135/) (2025)
- [Building Rearticulable Models for Arbitrary 3D Objects from 4D Point Clouds](https://mlsanity.com/paper/2306.00979/) (2023)
- [Understanding 3D Object Interaction from a Single Image](https://mlsanity.com/paper/2305.09664/) (2023)
- [Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Clouds](https://mlsanity.com/paper/2210.07442/) (2022)
- [DragAPart: Learning a Part-Level Motion Prior for Articulated Objects](https://mlsanity.com/paper/2403.15382/) (2024)
- [Articraft: An Agentic System for Scalable Articulated 3D Asset Generation](https://mlsanity.com/paper/2605.15187/) (2026)
- [DRAWER: Digital Reconstruction and Articulation With Environment Realism](https://mlsanity.com/paper/2504.15278/) (2025)

## Abstract (arXiv)

Digitizing physical objects into the virtual world has the potential to unlock new research and applications in embodied AI and mixed reality. This work focuses on recreating interactive digital twins of real-world articulated objects, which can be directly imported into virtual environments. We introduce Ditto to learn articulation model estimation and 3D geometry reconstruction of an articulated object through interactive perception. Given a pair of visual observations of an articulated object before and after interaction, Ditto reconstructs part-level geometry and estimates the articulation model of the object. We employ implicit neural representations for joint geometry and articulation modeling. Our experiments show that Ditto effectively builds digital twins of articulated objects in a category-agnostic way. We also apply Ditto to real-world objects and deploy the recreated digital twins in physical simulation. Code and additional results are available at https://ut-austin-rpl.github.io/Ditto