# Watch It Move: Unsupervised Discovery of 3D Joints for Re-Posing of Articulated Objects

arXiv: https://arxiv.org/abs/2112.11347 · HTML: https://mlsanity.com/paper/2112.11347/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 79 references that are in the map.
- First posted on arXiv 21 Dec 2021.

## Builds on (most cited first, 30 of 61)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image](https://mlsanity.com/paper/1607.08128/) (2016)
- [UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://mlsanity.com/paper/2104.10078/) (2021)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video](https://mlsanity.com/paper/2012.12247/) (2020)
- [Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies](https://mlsanity.com/paper/2105.02872/) (2021)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2011.14672/) (2020)
- [Integral Human Pose Regression](https://mlsanity.com/paper/1711.08229/) (2017)
- [Pose Guided Person Image Generation](https://mlsanity.com/paper/1705.09368/) (2017)
- [Learning Shape Templates with Structured Implicit Functions](https://mlsanity.com/paper/1904.06447/) (2019)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)
- [A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose](https://mlsanity.com/paper/2102.06199/) (2021)
- [HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields](https://mlsanity.com/paper/2106.13228/) (2021)
- [Stereo Magnification: Learning View Synthesis using Multiplane Images](https://mlsanity.com/paper/1805.09817/) (2018)
- [Neural Articulated Radiance Field](https://mlsanity.com/paper/2104.03110/) (2021)
- [Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision](https://mlsanity.com/paper/1905.06817/) (2019)
- [Learnable Triangulation of Human Pose](https://mlsanity.com/paper/1905.05754/) (2019)
- [Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis](https://mlsanity.com/paper/1909.12224/) (2019)

## Built on by (most cited first, 30 of 50)

- [Deforming Radiance Fields with Cages](https://mlsanity.com/paper/2207.12298/) (2022)
- [ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splatting](https://mlsanity.com/paper/2502.19459/) (2025)
- [Unsupervised Learning of Efficient Geometry-Aware Neural Articulated Representations](https://mlsanity.com/paper/2204.08839/) (2022)
- [CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objects](https://mlsanity.com/paper/2303.15782/) (2023)
- [Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction](https://mlsanity.com/paper/2409.18121/) (2024)
- [NeuPhysics: Editable Neural Geometry and Physics from Monocular Videos](https://mlsanity.com/paper/2210.12352/) (2022)
- [Reconstructing Animatable Categories from Videos](https://mlsanity.com/paper/2305.06351/) (2023)
- [PartNeRF: Generating Part-Aware Editable 3D Shapes without 3D Supervision](https://mlsanity.com/paper/2303.09554/) (2023)
- [CASA: Category-agnostic Skeletal Animal Reconstruction](https://mlsanity.com/paper/2211.03568/) (2022)
- [Building Rearticulable Models for Arbitrary 3D Objects from 4D Point Clouds](https://mlsanity.com/paper/2306.00979/) (2023)
- [DRAWER: Digital Reconstruction and Articulation With Environment Realism](https://mlsanity.com/paper/2504.15278/) (2025)
- [AnimatableDreamer: Text-Guided Non-rigid 3D Model Generation and Reconstruction with Canonical Score Distillation](https://mlsanity.com/paper/2312.03795/) (2023)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [HiMoR: Monocular Deformable Gaussian Reconstruction with Hierarchical Motion Representation](https://mlsanity.com/paper/2504.06210/) (2025)
- [AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints](https://mlsanity.com/paper/2205.10636/) (2022)
- [Template-free Articulated Neural Point Clouds for Reposable View Synthesis](https://mlsanity.com/paper/2305.19065/) (2023)
- [VideoArtGS: Building Digital Twins of Articulated Objects from Monocular Video](https://mlsanity.com/paper/2509.17647/) (2025)
- [Survey on Modeling of Human-made Articulated Objects](https://mlsanity.com/paper/2403.14937/) (2024)
- [Zero-shot Pose Transfer for Unrigged Stylized 3D Characters](https://mlsanity.com/paper/2306.00200/) (2023)
- [Template-free Articulated Gaussian Splatting for Real-time Reposable Dynamic View Synthesis](https://mlsanity.com/paper/2412.05570/) (2024)
- [Detection Based Part-level Articulated Object Reconstruction from Single RGBD Image](https://mlsanity.com/paper/2504.03177/) (2025)
- [MovingParts: Motion-based 3D Part Discovery in Dynamic Radiance Field](https://mlsanity.com/paper/2303.05703/) (2023)
- [S3O: A Dual-Phase Approach for Reconstructing Dynamic Shape and Skeleton of Articulated Objects from Single Monocular Video](https://mlsanity.com/paper/2405.12607/) (2024)
- [Inferring Articulated Rigid Body Dynamics from RGBD Video](https://mlsanity.com/paper/2203.10488/) (2022)
- [CAMM: Building Category-Agnostic and Animatable 3D Models from Monocular Videos](https://mlsanity.com/paper/2304.06937/) (2023)
- [3D Human Keypoints Estimation From Point Clouds in the Wild Without Human Labels](https://mlsanity.com/paper/2306.04745/) (2023)
- [Learning Implicit Representation for Reconstructing Articulated Objects](https://mlsanity.com/paper/2401.08809/) (2024)
- [Articulated 3D Human-Object Interactions from RGB Videos: An Empirical Analysis of Approaches and Challenges](https://mlsanity.com/paper/2209.05612/) (2022)
- [Predict-Optimize-Distill: A Self-Improving Cycle for 4D Object Understanding](https://mlsanity.com/paper/2504.17441/) (2025)
- [BlendFields: Few-Shot Example-Driven Facial Modeling](https://mlsanity.com/paper/2305.07514/) (2023)

## Abstract (arXiv)

Rendering articulated objects while controlling their poses is critical to applications such as virtual reality or animation for movies. Manipulating the pose of an object, however, requires the understanding of its underlying structure, that is, its joints and how they interact with each other. Unfortunately, assuming the structure to be known, as existing methods do, precludes the ability to work on new object categories. We propose to learn both the appearance and the structure of previously unseen articulated objects by observing them move from multiple views, with no joints annotation supervision, or information about the structure. We observe that 3D points that are static relative to one another should belong to the same part, and that adjacent parts that move relative to each other must be connected by a joint. To leverage this insight, we model the object parts in 3D as ellipsoids, which allows us to identify joints. We combine this explicit representation with an implicit one that compensates for the approximation introduced. We show that our method works for different structures, from quadrupeds, to single-arm robots, to humans.