# iTACO: Interactable Digital Twins of Articulated Objects from Casually Captured RGBD Videos

arXiv: https://arxiv.org/abs/2506.08334 · HTML: https://mlsanity.com/paper/2506.08334/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 68 of its 85 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Builds on (most cited first, 30 of 68)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [4D Gaussian Splatting for Real-Time Dynamic Scene Rendering](https://mlsanity.com/paper/2310.08528/) (2023)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [Continuous 3D Perception Model with Persistent State](https://mlsanity.com/paper/2501.12387/) (2025)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting](https://mlsanity.com/paper/2310.10642/) (2023)
- [MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion](https://mlsanity.com/paper/2410.03825/) (2024)
- [Shape of Motion: 4D Reconstruction from a Single Video](https://mlsanity.com/paper/2407.13764/) (2024)
- [MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos](https://mlsanity.com/paper/2412.04463/) (2024)
- [BANMo: Building Animatable 3D Neural Models from Many Casual Videos](https://mlsanity.com/paper/2112.12761/) (2021)
- [4D-Rotor Gaussian Splatting: Towards Efficient Novel View Synthesis for Dynamic Scenes](https://mlsanity.com/paper/2402.03307/) (2024)
- [Robust Consistent Video Depth Estimation](https://mlsanity.com/paper/2012.05901/) (2020)
- [Category-Level Articulated Object Pose Estimation](https://mlsanity.com/paper/1912.11913/) (2019)
- [Ditto: Building Digital Twins of Articulated Objects from Interaction](https://mlsanity.com/paper/2202.08227/) (2022)
- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation](https://mlsanity.com/paper/2403.03949/) (2024)
- [Shape2Motion: Joint Analysis of Motion Parts and Attributes from 3D Shapes](https://mlsanity.com/paper/1903.03911/) (2019)
- [PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objects](https://mlsanity.com/paper/2308.07391/) (2023)
- [LASR: Learning Articulated Shape Reconstruction from a Monocular Video](https://mlsanity.com/paper/2105.02976/) (2021)
- [A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation](https://mlsanity.com/paper/2104.07645/) (2021)
- [URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images](https://mlsanity.com/paper/2405.11656/) (2024)
- [Neural Kernel Surface Reconstruction](https://mlsanity.com/paper/2305.19590/) (2023)
- [SAMPart3D: Segment Any Part in 3D Objects](https://mlsanity.com/paper/2411.07184/) (2024)

## Built on by (most cited first, 19 of 19)

- [DreamArt: Generating Interactable Articulated Objects from a Single Image](https://mlsanity.com/paper/2507.05763/) (2025)
- [VideoArtGS: Building Digital Twins of Articulated Objects from Monocular Video](https://mlsanity.com/paper/2509.17647/) (2025)
- [FunRec: Reconstructing Functional 3D Scenes from Egocentric Interaction Videos](https://mlsanity.com/paper/2604.05621/) (2026)
- [REACT3D: Recovering Articulations for Interactive Physical 3D Scenes](https://mlsanity.com/paper/2510.11340/) (2025)
- [Articulation in Motion: Prior-free Part Mobility Analysis for Articulated Objects By Dynamic-Static Disentanglement](https://mlsanity.com/paper/2603.02910/) (2026)
- [ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions](https://mlsanity.com/paper/2603.25791/) (2026)
- [Articulat3D: Reconstructing Articulated Digital Twins From Monocular Videos with Geometric and Motion Constraints](https://mlsanity.com/paper/2603.11606/) (2026)
- [FreeArtGS: Articulated Gaussian Splatting Under Free-moving Scenario](https://mlsanity.com/paper/2603.22102/) (2026)
- [sim2art: Accurate Articulated Object Modeling from a Single Video using Synthetic Training Data Only](https://mlsanity.com/paper/2512.07698/) (2025)
- [PAWS: Perception of Articulation in the Wild at Scale from Egocentric Videos](https://mlsanity.com/paper/2603.25539/) (2026)
- [EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templates](https://mlsanity.com/paper/2604.11038/) (2026)
- [Articulation in Prime: Primitive-Based Articulated Object Understanding from a Single Casual Video](https://mlsanity.com/paper/2605.18645/) (2026)
- [AgentSTAR: Agentic Shape Tracking and Reconstruction from Monocular Videos](https://mlsanity.com/paper/2609.24487/) (2026)
- [PWM-ArtGen: Part World Model for Articulated Object Generation](https://mlsanity.com/paper/2607.02045/) (2026)
- [Track, Articulate, Act: Generating Articulation from Casual Human Videos](https://mlsanity.com/paper/2609.19119/) (2026)
- [FunArt: Decoding Functional Structure and Articulation from Generative 3D Latents](https://mlsanity.com/paper/2609.20673/) (2026)
- [FAMOS: Feed-Forward 3D Articulation Modeling from Sparse Observations](https://mlsanity.com/paper/2609.20817/) (2026)
- [Track2Art: Articulated Object Model Recovery with Visual-Geometric Track Representations](https://mlsanity.com/paper/2609.27675/) (2026)
- [MoSE3: Learning World-Space SE(3) at Every Pixel](https://mlsanity.com/paper/2610.03716/) (2026)

## Abstract (arXiv)

Articulated objects are prevalent in daily life. Interactable digital twins of such objects have numerous applications in embodied AI and robotics. Unfortunately, current methods to digitize articulated real-world objects require carefully captured data, preventing practical, scalable, and generalizable acquisition. We focus on motion analysis and part-level segmentation of an articulated object from a casually captured RGBD video shot with a hand-held camera. A casually captured video of an interaction with an articulated object is easy to obtain at scale using smartphones. However, this setting is challenging due to simultaneous object and camera motion and significant occlusions as the person interacts with the object. To tackle these challenges, we introduce iTACO: a coarse-to-fine framework that infers joint parameters and segments movable parts of the object from a dynamic RGBD video. To evaluate our method under this new setting, we build a dataset of 784 videos containing 284 objects across 11 categories that is 20× larger than available in prior work. We then compare our approach with existing methods that also take video as input. Our experiments show that iTACO outperforms existing articulated object digital twin methods on both synthetic and real casually captured RGBD videos.