# Act the Part: Learning Interaction Strategies for Articulated Object Part Discovery

arXiv: https://arxiv.org/abs/2105.01047 · HTML: https://mlsanity.com/paper/2105.01047/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 56 references that are in the map.
- First posted on arXiv 3 May 2021.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [Where2Act: From Pixels to Actions for Articulated 3D Objects](https://mlsanity.com/paper/2101.02692/) (2021)
- [Category-Level Articulated Object Pose Estimation](https://mlsanity.com/paper/1912.11913/) (2019)
- [Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning](https://mlsanity.com/paper/1803.09956/) (2018)
- [Unsupervised Learning of Object Landmarks through Conditional Image Generation](https://mlsanity.com/paper/1806.07823/) (2018)
- [SCOPS: Self-Supervised Co-Part Segmentation](https://mlsanity.com/paper/1905.01298/) (2019)
- [Interactive Perception: Leveraging Action in Perception and Perception in Action](https://mlsanity.com/paper/1604.03670/) (2016)
- [Deep Part Induction from Articulated Object Pairs](https://mlsanity.com/paper/1809.07417/) (2018)
- [DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactions](https://mlsanity.com/paper/1906.03853/) (2019)
- [SE3-Nets: Learning Rigid Body Motion using Deep Neural Networks](https://mlsanity.com/paper/1606.02378/) (2016)
- [Camera-to-Robot Pose Estimation from a Single Image](https://mlsanity.com/paper/1911.09231/) (2019)
- [MultiBodySync: Multi-Body Segmentation and Motion Estimation via 3D Scan Synchronization](https://mlsanity.com/paper/2101.06605/) (2021)
- [A Probabilistic Framework for Learning Kinematic Models of Articulated Objects](https://mlsanity.com/paper/1405.7705/) (2014)
- [Learning 3D Dynamic Scene Representations for Robot Manipulation](https://mlsanity.com/paper/2011.01968/) (2020)
- [The Curious Robot: Learning Visual Representations via Physical Interactions](https://mlsanity.com/paper/1604.01360/) (2016)
- [Nothing But Geometric Constraints: A Model-Free Method for Articulated Object Pose Estimation](https://mlsanity.com/paper/2012.00088/) (2020)
- [Semantic Part Segmentation using Compositional Model combining Shape and Appearance](https://mlsanity.com/paper/1412.6124/) (2014)
- [3D-OES: Viewpoint-Invariant Object-Factorized Environment Simulators](https://mlsanity.com/paper/2011.06464/) (2020)
- [Unsupervised Discovery of Parts, Structure, and Dynamics](https://mlsanity.com/paper/1903.05136/) (2019)
- [Learning Instance Segmentation by Interaction](https://mlsanity.com/paper/1806.08354/) (2018)

## Built on by (most cited first, 30 of 45)

- [Ditto: Building Digital Twins of Articulated Objects from Interaction](https://mlsanity.com/paper/2202.08227/) (2022)
- [GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts](https://mlsanity.com/paper/2211.05272/) (2022)
- [PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objects](https://mlsanity.com/paper/2308.07391/) (2023)
- [VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects](https://mlsanity.com/paper/2106.14440/) (2021)
- [Real2Code: Reconstruct Articulated Objects via Code Generation](https://mlsanity.com/paper/2406.08474/) (2024)
- [ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splatting](https://mlsanity.com/paper/2502.19459/) (2025)
- [DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects](https://mlsanity.com/paper/2305.05706/) (2023)
- [UMPNet: Universal Manipulation Policy Network for Articulated Objects](https://mlsanity.com/paper/2109.05668/) (2021)
- [Unsupervised Part Discovery from Contrastive Reconstruction](https://mlsanity.com/paper/2111.06349/) (2021)
- [3D-Aware Object Goal Navigation via Simultaneous Exploration and Identification](https://mlsanity.com/paper/2212.00338/) (2022)
- [Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modeling](https://mlsanity.com/paper/2502.02590/) (2025)
- [Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perception](https://mlsanity.com/paper/2302.01295/) (2023)
- [Continuous Scene Representations for Embodied AI](https://mlsanity.com/paper/2203.17251/) (2022)
- [EmbodiedGen: Towards a Generative 3D World Engine for Embodied Intelligence](https://mlsanity.com/paper/2506.10600/) (2025)
- [Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated Objects](https://mlsanity.com/paper/2309.07473/) (2023)
- [DragAPart: Learning a Part-Level Motion Prior for Articulated Objects](https://mlsanity.com/paper/2403.15382/) (2024)
- [Learning Foresightful Dense Visual Affordance for Deformable Object Manipulation](https://mlsanity.com/paper/2303.11057/) (2023)
- [Learning Environment-Aware Affordance for 3D Articulated Object Manipulation under Occlusions](https://mlsanity.com/paper/2309.07510/) (2023)
- [DIPO: Dual-State Images Controlled Articulated Object Generation Powered by Diverse Data](https://mlsanity.com/paper/2505.20460/) (2025)
- [DualAfford: Learning Collaborative Visual Affordance for Dual-gripper Manipulation](https://mlsanity.com/paper/2207.01971/) (2022)
- [Learning Category-Level Generalizable Object Manipulation Policy via Generative Adversarial Self-Imitation Learning from Demonstrations](https://mlsanity.com/paper/2203.02107/) (2022)
- [VideoArtGS: Building Digital Twins of Articulated Objects from Monocular Video](https://mlsanity.com/paper/2509.17647/) (2025)
- [Survey on Modeling of Human-made Articulated Objects](https://mlsanity.com/paper/2403.14937/) (2024)
- [SplArt: Articulation Estimation and Part-Level Reconstruction with 3D Gaussian Splatting](https://mlsanity.com/paper/2506.03594/) (2025)
- [Robot Skill Adaptation via Soft Actor-Critic Gaussian Mixture Models](https://mlsanity.com/paper/2111.13129/) (2021)
- [AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding](https://mlsanity.com/paper/2606.06155/) (2026)
- [STRAP: Structured Object Affordance Segmentation with Point Supervision](https://mlsanity.com/paper/2304.08492/) (2023)
- [IFR-Explore: Learning Inter-object Functional Relationships in 3D Indoor Scenes](https://mlsanity.com/paper/2112.05298/) (2021)
- [GarmentPile: Point-Level Visual Affordance Guided Retrieval and Adaptation for Cluttered Garments Manipulation](https://mlsanity.com/paper/2503.09243/) (2025)
- [ScrewSplat: An End-to-End Method for Articulated Object Recognition](https://mlsanity.com/paper/2508.02146/) (2025)

## Abstract (arXiv)

People often use physical intuition when manipulating articulated objects, irrespective of object semantics. Motivated by this observation, we identify an important embodied task where an agent must play with objects to recover their parts. To this end, we introduce Act the Part (AtP) to learn how to interact with articulated objects to discover and segment their pieces. By coupling action selection and motion segmentation, AtP is able to isolate structures to make perceptual part recovery possible without semantic labels. Our experiments show AtP learns efficient strategies for part discovery, can generalize to unseen categories, and is capable of conditional reasoning for the task. Although trained in simulation, we show convincing transfer to real world data with no fine-tuning.