mlsanityData to 7 Oct 2026

Structure from Action: Learning Interactions for Articulated Object 3D Structure Discovery

arXiv 2207.08997 ↗First posted 19 Jul 2022cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

We introduce Structure from Action (SfA), a framework to discover 3D part geometry and joint parameters of unseen articulated objects via a sequence of inferred interactions. Our key insight is that 3D interaction and perception should be considered in conjunction to construct 3D articulated CAD models, especially for categories not seen during training. By selecting informative interactions, SfA discovers parts and reveals occluded surfaces, like the inside of a closed drawer. By aggregating visual observations in 3D, SfA accurately segments multiple parts, reconstructs part geometry, and infers all joint parameters in a canonical coordinate frame. Our experiments demonstrate that a SfA model trained in simulation can generalize to many unseen object categories with diverse structures and to real-world objects. Empirically, SfA outperforms a pipeline of state-of-the-art components by 25.4 3D IoU percentage points on unseen categories, while matching already performant joint estimation baselines.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 43 of its 57 references that are in the map. First posted on arXiv 19 Jul 2022.

Builds on

43 of 57 references.

References · 43

  1. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  3. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  4. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  5. 2020Point Transformercited by 1,462
  6. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  7. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  8. 2021Motion Representations for Articulated Animationcited by 253
  9. 2021Neural Articulated Radiance Fieldcited by 196
  10. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  11. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 179
  12. 2019Category-Level Articulated Object Pose Estimationcited by 158
31 more
  1. 2022Ditto: Building Digital Twins of Articulated Objects from Interactioncited by 152
  2. 2021A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representationcited by 113
  3. 2019SCOPS: Self-Supervised Co-Part Segmentationcited by 100
  4. 2022FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objectscited by 91
  5. 2021CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Cloudscited by 83
  6. 2022AKB-48: A Real-World Articulated Object Knowledge Basecited by 81
  7. 2021VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objectscited by 78
  8. 2020ScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theorycited by 70
  9. 2018Deep Part Induction from Articulated Object Pairscited by 66
  10. 2021UMPNet: Universal Manipulation Policy Network for Articulated Objectscited by 64
  11. 2021Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objectscited by 50
  12. 2019Camera-to-Robot Pose Estimation from a Single Imagecited by 48
  13. 2021MultiBodySync: Multi-Body Segmentation and Motion Estimation via 3D Scan Synchronizationcited by 46
  14. 2021Act the Part: Learning Interaction Strategies for Articulated Object Part Discoverycited by 45
  15. 2021Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulationcited by 44
  16. 2014A Probabilistic Framework for Learning Kinematic Models of Articulated Objectscited by 40
  17. 2020Visual Identification of Articulated Object Partscited by 36
  18. 2022Understanding 3D Object Articulation in Internet Videoscited by 36
  19. 2018The RBO Dataset of Articulated Objects and Interactionscited by 34
  20. 2020Nothing But Geometric Constraints: A Model-Free Method for Articulated Object Pose Estimationcited by 25
  21. 2014Semantic Part Segmentation using Compositional Model combining Shape and Appearancecited by 22
  22. 2019Unsupervised Discovery of Parts, Structure, and Dynamicscited by 20
  23. 2021Distributional Depth-Based Estimation of Object Articulation Modelscited by 18
  24. 2015Learning Articulated Motions From Visual Demonstrationcited by 17
  25. 2021SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learningcited by 16
  26. 2022Learning Category-Level Generalizable Object Manipulation Policy via Generative Adversarial Self-Imitation Learning from Demonstrationscited by 15
  27. 2021Learning to Infer Kinematic Hierarchies for Novel Object Instancescited by 10
  28. 2018On the manipulation of articulated objects in human-robot cooperation scenarioscited by 7
  29. 2020Manipulation of Articulated Objects using Dual-arm Robots via Answer Set Programmingcited by 7
  30. 2021V-MAO: Generative Modeling for Multi-Arm Manipulation of Articulated Objectscited by 3
  31. 2015Deep Learning for Semantic Part Segmentation with High-Level Guidancecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modelingcited by 41
  2. 2024Survey on Modeling of Human-made Articulated Objectscited by 14
  3. 2025Part Segmentation and Motion Estimation for Articulated Objects with Dynamic 3D Gaussianscited by 5
  4. 2026Online Estimation and Manipulation of Articulated Objectscited by 3