# Understanding 3D Object Articulation in Internet Videos

arXiv: https://arxiv.org/abs/2203.16531 · HTML: https://mlsanity.com/paper/2203.16531/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 73 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Builds on (most cited first, 30 of 52)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [PARE: Part Attention Regressor for 3D Human Body Estimation](https://mlsanity.com/paper/2104.08527/) (2021)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [Detecting and Recognizing Human-Object Interactions](https://mlsanity.com/paper/1704.07333/) (2017)
- [Where2Act: From Pixels to Actions for Articulated 3D Objects](https://mlsanity.com/paper/2101.02692/) (2021)
- [Learning a Predictable and Generative Vector Representation for Objects](https://mlsanity.com/paper/1603.08637/) (2016)
- [Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image](https://mlsanity.com/paper/2002.12212/) (2020)
- [Category-Level Articulated Object Pose Estimation](https://mlsanity.com/paper/1912.11913/) (2019)
- [Rearrangement: A Challenge for Embodied AI](https://mlsanity.com/paper/2011.01975/) (2020)
- [Shape2Motion: Joint Analysis of Motion Parts and Attributes from 3D Shapes](https://mlsanity.com/paper/1903.03911/) (2019)
- [A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation](https://mlsanity.com/paper/2104.07645/) (2021)
- [Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimation](https://mlsanity.com/paper/2004.03686/) (2020)
- [PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Image](https://mlsanity.com/paper/1812.04072/) (2018)

## Built on by (most cited first, 30 of 36)

- [GenAD: Generalized Predictive Model for Autonomous Driving](https://mlsanity.com/paper/2403.09630/) (2024)
- [URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images](https://mlsanity.com/paper/2405.11656/) (2024)
- [Full-Body Articulated Human-Object Interaction](https://mlsanity.com/paper/2212.10621/) (2022)
- [OPDMulti: Openable Part Detection for Multiple Objects](https://mlsanity.com/paper/2303.14087/) (2023)
- [Building Rearticulable Models for Arbitrary 3D Objects from 4D Point Clouds](https://mlsanity.com/paper/2306.00979/) (2023)
- [Understanding 3D Object Interaction from a Single Image](https://mlsanity.com/paper/2305.09664/) (2023)
- [DRAWER: Digital Reconstruction and Articulation With Environment Realism](https://mlsanity.com/paper/2504.15278/) (2025)
- [Articulate your NeRF: Unsupervised articulated object modeling via conditional view synthesis](https://mlsanity.com/paper/2406.16623/) (2024)
- [S2O: Static to Openable Enhancement for Articulated 3D Objects](https://mlsanity.com/paper/2409.18896/) (2024)
- [FreeArt3D: Training-Free Articulated Object Generation using 3D Diffusion](https://mlsanity.com/paper/2510.25765/) (2025)
- [GaussianArt: Unified Modeling of Geometry and Motion for Articulated Objects](https://mlsanity.com/paper/2508.14891/) (2025)
- [LEIA: Latent View-invariant Embeddings for Implicit 3D Articulation](https://mlsanity.com/paper/2409.06703/) (2024)
- [Exploring Low-Dimensional Subspaces in Diffusion Models for Controllable Image Editing](https://mlsanity.com/paper/2409.02374/) (2024)
- [Survey on Modeling of Human-made Articulated Objects](https://mlsanity.com/paper/2403.14937/) (2024)
- [LARM: A Large Articulated-Object Reconstruction Model](https://mlsanity.com/paper/2511.11563/) (2025)
- [IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments](https://mlsanity.com/paper/2504.06827/) (2025)
- [RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Grasping](https://mlsanity.com/paper/2507.23734/) (2025)
- [Articulate3D: Holistic Understanding of 3D Scenes as Universal Scene Description](https://mlsanity.com/paper/2412.01398/) (2024)
- [2HandedAfforder: Learning Precise Actionable Bimanual Affordances from Human Videos](https://mlsanity.com/paper/2503.09320/) (2025)
- [ART: Articulated Reconstruction Transformer](https://mlsanity.com/paper/2512.14671/) (2025)
- [Articulated 3D Human-Object Interactions from RGB Videos: An Empirical Analysis of Approaches and Challenges](https://mlsanity.com/paper/2209.05612/) (2022)
- [Neural Shape Compiler: A Unified Framework for Transforming between Text, Point Cloud, and Program](https://mlsanity.com/paper/2212.12952/) (2022)
- [RoSI: Recovering 3D Shape Interiors from Few Articulation Images](https://mlsanity.com/paper/2304.06342/) (2023)
- [MonoMobility: Zero-Shot 3D Mobility Analysis from Monocular Videos](https://mlsanity.com/paper/2505.11868/) (2025)
- [Articulation in Motion: Prior-free Part Mobility Analysis for Articulated Objects By Dynamic-Static Disentanglement](https://mlsanity.com/paper/2603.02910/) (2026)
- [Structure from Action: Learning Interactions for Articulated Object 3D Structure Discovery](https://mlsanity.com/paper/2207.08997/) (2022)
- [FreeArtGS: Articulated Gaussian Splatting Under Free-moving Scenario](https://mlsanity.com/paper/2603.22102/) (2026)
- [DYNAMO: Dependency-Aware Deep Learning Framework for Articulated Assembly Motion Prediction](https://mlsanity.com/paper/2509.12430/) (2025)
- [PAWS: Perception of Articulation in the Wild at Scale from Egocentric Videos](https://mlsanity.com/paper/2603.25539/) (2026)
- [PAOLI: Pose-free Articulated Object Learning from Sparse-view Images](https://mlsanity.com/paper/2509.04276/) (2025)

## Abstract (arXiv)

We propose to investigate detecting and characterizing the 3D planar articulation of objects from ordinary videos. While seemingly easy for humans, this problem poses many challenges for computers. We propose to approach this problem by combining a top-down detection system that finds planes that can be articulated along with an optimization approach that solves for a 3D plane that can explain a sequence of observed articulations. We show that this system can be trained on a combination of videos and 3D scan datasets. When tested on a dataset of challenging Internet videos and the Charades dataset, our approach obtains strong performance. Project site: https://jasonqsy.github.io/Articulation3D