# Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perception

arXiv: https://arxiv.org/abs/2302.01295 · HTML: https://mlsanity.com/paper/2302.01295/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 2 Feb 2023.

## Builds on (most cited first, 26 of 26)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Neural Articulated Radiance Field](https://mlsanity.com/paper/2104.03110/) (2021)
- [iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks](https://mlsanity.com/paper/2108.03272/) (2021)
- [Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimation](https://mlsanity.com/paper/1808.05942/) (2018)
- [Where2Act: From Pixels to Actions for Articulated 3D Objects](https://mlsanity.com/paper/2101.02692/) (2021)
- [Category-Level Articulated Object Pose Estimation](https://mlsanity.com/paper/1912.11913/) (2019)
- [Ditto: Building Digital Twins of Articulated Objects from Interaction](https://mlsanity.com/paper/2202.08227/) (2022)
- [ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging](https://mlsanity.com/paper/1904.06830/) (2019)
- [Grounded Human-Object Interaction Hotspots from Video](https://mlsanity.com/paper/1812.04558/) (2018)
- [AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection](https://mlsanity.com/paper/1709.07326/) (2017)
- [Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning](https://mlsanity.com/paper/1803.09956/) (2018)
- [Shape2Motion: Joint Analysis of Motion Parts and Attributes from 3D Shapes](https://mlsanity.com/paper/1903.03911/) (2019)
- [Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations](https://mlsanity.com/paper/2104.01542/) (2021)
- [3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding](https://mlsanity.com/paper/2103.16397/) (2021)
- [CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds](https://mlsanity.com/paper/2104.03437/) (2021)
- [Interactive Perception: Leveraging Action in Perception and Perception in Action](https://mlsanity.com/paper/1604.03670/) (2016)
- [ScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theory](https://mlsanity.com/paper/2008.10518/) (2020)
- [Learning Affordance Landscapes for Interaction Exploration in 3D Environments](https://mlsanity.com/paper/2008.09241/) (2020)
- [CubiCasa5K: A Dataset and an Improved Multi-Task Model for Floorplan Image Analysis](https://mlsanity.com/paper/1904.01920/) (2019)
- [Act the Part: Learning Interaction Strategies for Articulated Object Part Discovery](https://mlsanity.com/paper/2105.01047/) (2021)
- [A Probabilistic Framework for Learning Kinematic Models of Articulated Objects](https://mlsanity.com/paper/1405.7705/) (2014)
- [Learning Articulated Motions From Visual Demonstration](https://mlsanity.com/paper/1502.01659/) (2015)
- [SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning](https://mlsanity.com/paper/2111.14693/) (2021)
- [Action Image Representation: Learning Scalable Deep Grasping Policies with Zero Real World Data](https://mlsanity.com/paper/2005.06594/) (2020)
- [Learning to Infer Kinematic Hierarchies for Novel Object Instances](https://mlsanity.com/paper/2110.07911/) (2021)

## Built on by (most cited first, 30 of 41)

- [PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objects](https://mlsanity.com/paper/2308.07391/) (2023)
- [URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images](https://mlsanity.com/paper/2405.11656/) (2024)
- [Real2Code: Reconstruct Articulated Objects via Code Generation](https://mlsanity.com/paper/2406.08474/) (2024)
- [ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splatting](https://mlsanity.com/paper/2502.19459/) (2025)
- [AffordanceLLM: Grounding Affordance from Vision Language Models](https://mlsanity.com/paper/2401.06341/) (2024)
- [CAGE: Controllable Articulation GEneration](https://mlsanity.com/paper/2312.09570/) (2023)
- [RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation](https://mlsanity.com/paper/2402.15487/) (2024)
- [Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modeling](https://mlsanity.com/paper/2502.02590/) (2025)
- [PhysX-3D: Physical-Grounded 3D Asset Generation](https://mlsanity.com/paper/2507.12465/) (2025)
- [OPDMulti: Openable Part Detection for Multiple Objects](https://mlsanity.com/paper/2303.14087/) (2023)
- [Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated Objects](https://mlsanity.com/paper/2309.07473/) (2023)
- [Understanding 3D Object Interaction from a Single Image](https://mlsanity.com/paper/2305.09664/) (2023)
- [DRAWER: Digital Reconstruction and Articulation With Environment Realism](https://mlsanity.com/paper/2504.15278/) (2025)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [ASID: Active Exploration for System Identification in Robotic Manipulation](https://mlsanity.com/paper/2404.12308/) (2024)
- [X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real](https://mlsanity.com/paper/2505.07096/) (2025)
- [Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videos](https://mlsanity.com/paper/2502.09886/) (2025)
- [NAP: Neural 3D Articulation Prior](https://mlsanity.com/paper/2305.16315/) (2023)
- [The Reality Gap in Robotics: Challenges, Solutions, and Best Practices](https://mlsanity.com/paper/2510.20808/) (2025)
- [Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusions](https://mlsanity.com/paper/2309.07510/) (2023)
- [UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation](https://mlsanity.com/paper/2506.09284/) (2025)
- [S2O: Static to Openable Enhancement for Articulated 3D Objects](https://mlsanity.com/paper/2409.18896/) (2024)
- [Articulated Object Estimation in the Wild](https://mlsanity.com/paper/2509.01708/) (2025)
- [Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Reset](https://mlsanity.com/paper/2409.17126/) (2024)
- [VideoArtGS: Building Digital Twins of Articulated Objects from Monocular Video](https://mlsanity.com/paper/2509.17647/) (2025)
- [Survey on Modeling of Human-made Articulated Objects](https://mlsanity.com/paper/2403.14937/) (2024)
- [SplArt: Articulation Estimation and Part-Level Reconstruction with 3D Gaussian Splatting](https://mlsanity.com/paper/2506.03594/) (2025)
- [Grounding 3D Scene Affordance From Egocentric Interactions](https://mlsanity.com/paper/2409.19650/) (2024)
- [IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments](https://mlsanity.com/paper/2504.06827/) (2025)
- [RoomCraft: Controllable and Complete 3D Indoor Scene Generation](https://mlsanity.com/paper/2506.22291/) (2025)

## Abstract (arXiv)

Virtualizing the physical world into virtual models has been a critical technique for robot navigation and planning in the real world. To foster manipulation with articulated objects in everyday life, this work explores building articulation models of indoor scenes through a robot's purposeful interactions in these scenes. Prior work on articulation reasoning primarily focuses on siloed objects of limited categories. To extend to room-scale environments, the robot has to efficiently and effectively explore a large-scale 3D space, locate articulated objects, and infer their articulations. We introduce an interactive perception approach to this task. Our approach, named Ditto in the House, discovers possible articulated objects through affordance prediction, interacts with these objects to produce articulated motions, and infers the articulation properties from the visual observations before and after each interaction. It tightly couples affordance prediction and articulation inference to improve both tasks. We demonstrate the effectiveness of our approach in both simulation and real-world scenes. Code and additional results are available at https://ut-austin-rpl.github.io/HouseDitto/