mlsanityData to 7 Oct 2026

Ditto: Building Digital Twins of Articulated Objects from Interaction

arXiv 2202.08227 ↗First posted 16 Feb 2022cs.CV cs.AI cs.ROcited by 152

Not on the map. Why?

Abstract (arXiv)

Digitizing physical objects into the virtual world has the potential to unlock new research and applications in embodied AI and mixed reality. This work focuses on recreating interactive digital twins of real-world articulated objects, which can be directly imported into virtual environments. We introduce Ditto to learn articulation model estimation and 3D geometry reconstruction of an articulated object through interactive perception. Given a pair of visual observations of an articulated object before and after interaction, Ditto reconstructs part-level geometry and estimates the articulation model of the object. We employ implicit neural representations for joint geometry and articulation modeling. Our experiments show that Ditto effectively builds digital twins of articulated objects in a category-agnostic way. We also apply Ditto to real-world objects and deploy the recreated digital twins in physical simulation. Code and additional results are available at https://ut-austin-rpl.github.io/Ditto

Key facts. Cited by 152 papers in the map (data through 7 Oct 2026). Builds on 46 of its 63 references that are in the map. First posted on arXiv 16 Feb 2022.

Builds on

46 of 63 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  7. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  8. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  9. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  10. 2020Neural Sparse Voxel Fieldscited by 959
  11. 2017End-to-end Recovery of Human Shape and Posecited by 883
  12. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
34 more
  1. 2020Convolutional Occupancy Networkscited by 760
  2. 2018Learning Dexterous In-Hand Manipulationcited by 643
  3. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
  4. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  5. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  6. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  7. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  8. 2021Implicit Behavioral Cloningcited by 442
  9. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  10. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  11. 2018Learning to Estimate 3D Human Pose and Shape from a Single Color Imagecited by 226
  12. 2021Neural Articulated Radiance Fieldcited by 196
  13. 2021BANMo: Building Animatable 3D Neural Models from Many Casual Videoscited by 195
  14. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  15. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  16. 2018Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimationcited by 185
  17. 2019Category-Level Articulated Object Pose Estimationcited by 158
  18. 2019NASA: Neural Articulated Shape Approximationcited by 149
  19. 2018Scan2CAD: Learning CAD Model Alignment in RGB-D Scanscited by 142
  20. 2019Meta-Sim: Learning to Generate Synthetic Datasetscited by 128
  21. 2019Shape2Motion: Joint Analysis of Motion Parts and Attributes from 3D Shapescited by 117
  22. 2021LASR: Learning Articulated Shape Reconstruction from a Monocular Videocited by 115
  23. 2021A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representationcited by 113
  24. 2017MarrNet: 3D Shape Reconstruction via 2.5D Sketchescited by 108
  25. 2021Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representationscited by 107
  26. 2016Interactive Perception: Leveraging Action in Perception and Perception in Actioncited by 77
  27. 2020ScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theorycited by 70
  28. 2017MINOS: Multimodal Indoor Simulator for Navigation in Complex Environmentscited by 66
  29. 2018Deep Part Induction from Articulated Object Pairscited by 66
  30. 2020Neural Deformation Graphs for Globally-consistent Non-rigid Reconstructioncited by 66
  31. 2021Act the Part: Learning Interaction Strategies for Articulated Object Part Discoverycited by 45
  32. 2014A Probabilistic Framework for Learning Kinematic Models of Articulated Objectscited by 40
  33. 2020Nothing But Geometric Constraints: A Model-Free Method for Articulated Object Pose Estimationcited by 25
  34. 2021Distributional Depth-Based Estimation of Object Articulation Modelscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 17; 2024: 34; 2025: 48; 2026: 43.

152 papers.

Sort:

By area:

  1. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  2. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  3. 2023PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objectscited by 115
  4. 2022iSDF: Real-Time Neural Signed Distance Fields for Robot Perceptioncited by 107
  5. 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 106
  6. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 104
  7. 2024Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Modelcited by 90
  8. 2022Full-Body Articulated Human-Object Interactioncited by 76
  9. 2025ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splattingcited by 67
  10. 2024SINGAPO: Single Image Controlled Generation of Articulated Parts in Objectscited by 66
  11. 2023CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objectscited by 64
  12. 2023DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objectscited by 64
38 more
  1. 2024Real2Code: Reconstruct Articulated Objects via Code Generationcited by 64
  2. 2022SE(3)-Equivariant Relational Rearrangement with Neural Descriptor Fieldscited by 51
  3. 2023CAGE: Controllable Articulation GEnerationcited by 50
  4. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  5. 2024REACTO: Reconstructing Articulated Objects from a Single Videocited by 45
  6. 2025Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulationcited by 42
  7. 2025Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modelingcited by 41
  8. 2023Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perceptioncited by 40
  9. 2024MeshArt: Generating Articulated Meshes with Structure-Guided Transformerscited by 40
  10. 2023OPDMulti: Openable Part Detection for Multiple Objectscited by 38
  11. 2025ArticulatedGS: Self-supervised Digital Twin Modeling of Articulated Objects using 3D Gaussian Splattingcited by 35
  12. 2023Building Rearticulable Models for Arbitrary 3D Objects from 4D Point Cloudscited by 34
  13. 2023Understanding 3D Object Interaction from a Single Imagecited by 33
  14. 2023Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated Objectscited by 33
  15. 2024DragAPart: Learning a Part-Level Motion Prior for Articulated Objectscited by 32
  16. 2022Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Cloudscited by 31
  17. 2025DRAWER: Digital Reconstruction and Articulation With Environment Realismcited by 29
  18. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  19. 2024ASID: Active Exploration for System Identification in Robotic Manipulationcited by 27
  20. 2023Self-Supervised Category-Level Articulated Object Pose Estimation with Part-Level SE(3) Equivariancecited by 26
  21. 2023NAP: Neural 3D Articulation Priorcited by 25
  22. 2025DreamArt: Generating Interactable Articulated Objects from a Single Imagecited by 25
  23. 2026Articraft: An Agentic System for Scalable Articulated 3D Asset Generationcited by 25
  24. 2022Phone2Proc: Bringing Robust Robots Into Our Chaotic Worldcited by 24
  25. 2024Articulate your NeRF: Unsupervised articulated object modeling via conditional view synthesiscited by 24
  26. 2024PhysPart: Physically Plausible Part Completion for Interactable Objectscited by 24
  27. 2025PartNeXt: A Next-Generation Dataset for Fine-Grained and Hierarchical 3D Part Understandingcited by 24
  28. 2023Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusionscited by 23
  29. 2024Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representationscited by 23
  30. 2024S2O: Static to Openable Enhancement for Articulated 3D Objectscited by 22
  31. 2025FreeArt3D: Training-Free Articulated Object Generation using 3D Diffusioncited by 22
  32. 20233D Implicit Transporter for Temporally Consistent Keypoint Discoverycited by 20
  33. 2025Particulate: Feed-Forward 3D Object Articulationcited by 20
  34. 2023Banana: Banach Fixed-Point Network for Pointcloud Segmentation with Inter-Part Equivariancecited by 19
  35. 2025iTACO: Interactable Digital Twins of Articulated Objects from Casually Captured RGBD Videoscited by 18
  36. 2025Articulated Object Estimation in the Wildcited by 18
  37. 2025Infinite Mobility: Scalable High-Fidelity Synthesis of Articulated Objects via Procedural Generationcited by 17
  38. 2025FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulationcited by 17

50 most cited of 152 All 152 (JSON)