# Reconstructing Animatable Categories from Videos

arXiv: https://arxiv.org/abs/2305.06351 · HTML: https://mlsanity.com/paper/2305.06351/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 82 references that are in the map.
- First posted on arXiv 10 May 2023.

## Builds on (most cited first, 30 of 62)

- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [VIBE: Video Inference for Human Body Pose and Shape Estimation](https://mlsanity.com/paper/1912.05656/) (2019)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans](https://mlsanity.com/paper/2110.04994/) (2021)
- [HuMoR: 3D Human Motion Model for Robust Pose Estimation](https://mlsanity.com/paper/2105.04668/) (2021)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [3D Menagerie: Modeling the 3D shape and pose of animals](https://mlsanity.com/paper/1611.07700/) (2016)
- [ICON: Implicit Clothed humans Obtained from Normals](https://mlsanity.com/paper/2112.09127/) (2021)
- [What Do Single-view 3D Reconstruction Networks Learn?](https://mlsanity.com/paper/1905.03678/) (2019)
- [A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose](https://mlsanity.com/paper/2102.06199/) (2021)
- [PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction](https://mlsanity.com/paper/2007.03858/) (2020)
- [HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields](https://mlsanity.com/paper/2106.13228/) (2021)
- [BANMo: Building Animatable 3D Neural Models from Many Casual Videos](https://mlsanity.com/paper/2112.12761/) (2021)
- [Neural Articulated Radiance Field](https://mlsanity.com/paper/2104.03110/) (2021)

## Built on by (most cited first, 30 of 47)

- [GART: Gaussian Articulated Template Models](https://mlsanity.com/paper/2311.16099/) (2023)
- [GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation](https://mlsanity.com/paper/2403.12365/) (2024)
- [ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splatting](https://mlsanity.com/paper/2502.19459/) (2025)
- [4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models](https://mlsanity.com/paper/2406.07472/) (2024)
- [Learning the 3D Fauna of the Web](https://mlsanity.com/paper/2401.02400/) (2024)
- [REACTO: Reconstructing Articulated Objects from a Single Video](https://mlsanity.com/paper/2404.11151/) (2024)
- [Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfels](https://mlsanity.com/paper/2405.16822/) (2024)
- [DynPoint: Dynamic Neural Point For View Synthesis](https://mlsanity.com/paper/2310.18999/) (2023)
- [AnimatableDreamer: Text-Guided Non-rigid 3D Model Generation and Reconstruction with Canonical Score Distillation](https://mlsanity.com/paper/2312.03795/) (2023)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [Animal Avatars: Reconstructing Animatable 3D Animals from Casual Videos](https://mlsanity.com/paper/2403.17103/) (2024)
- [DRiVE: Diffusion-based Rigging Empowers Generation of Versatile and Expressive Characters](https://mlsanity.com/paper/2411.17423/) (2024)
- [InterTrack: Tracking Human Object Interaction without Object Templates](https://mlsanity.com/paper/2408.13953/) (2024)
- [ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image Collections](https://mlsanity.com/paper/2306.04619/) (2023)
- [Ponymation: Learning Articulated 3D Animal Motions from Unlabeled Online Videos](https://mlsanity.com/paper/2312.13604/) (2023)
- [Diffusion Priors for Dynamic View Synthesis from Monocular Videos](https://mlsanity.com/paper/2401.05583/) (2024)
- [VideoArtGS: Building Digital Twins of Articulated Objects from Monocular Video](https://mlsanity.com/paper/2509.17647/) (2025)
- [RigGS: Rigging of 3D Gaussians for Modeling Articulated Objects in Videos](https://mlsanity.com/paper/2503.16822/) (2025)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [A Survey on 3D Human Avatar Modeling -- From Reconstruction to Generation](https://mlsanity.com/paper/2406.04253/) (2024)
- [Articulated Kinematics Distillation from Video Diffusion Models](https://mlsanity.com/paper/2504.01204/) (2025)
- [DreaMo: Articulated 3D Reconstruction From A Single Casual Video](https://mlsanity.com/paper/2312.02617/) (2023)
- [AWOL: Analysis WithOut synthesis using Language](https://mlsanity.com/paper/2404.03042/) (2024)
- [S3O: A Dual-Phase Approach for Reconstructing Dynamic Shape and Skeleton of Articulated Objects from Single Monocular Video](https://mlsanity.com/paper/2405.12607/) (2024)
- [Sync4D: Video Guided Controllable Dynamics for Physics-Based 4D Generation](https://mlsanity.com/paper/2405.16849/) (2024)
- [DressRecon: Freeform 4D Human Reconstruction from Monocular Video](https://mlsanity.com/paper/2409.20563/) (2024)
- [DualPM: Dual Posed-Canonical Point Maps for 3D Shape and Pose Reconstruction](https://mlsanity.com/paper/2412.04464/) (2024)
- [RHOBIN Challenge: Reconstruction of Human Object Interaction](https://mlsanity.com/paper/2401.04143/) (2024)
- [Learning Implicit Representation for Reconstructing Articulated Objects](https://mlsanity.com/paper/2401.08809/) (2024)
- [ART: Articulated Reconstruction Transformer](https://mlsanity.com/paper/2512.14671/) (2025)

## Abstract (arXiv)

Building animatable 3D models is challenging due to the need for 3D scans, laborious registration, and manual rigging, which are difficult to scale to arbitrary categories. Recently, differentiable rendering provides a pathway to obtain high-quality 3D models from monocular videos, but these are limited to rigid categories or single instances. We present RAC that builds category 3D models from monocular videos while disentangling variations over instances and motion over time. Three key ideas are introduced to solve this problem: (1) specializing a skeleton to instances via optimization, (2) a method for latent space regularization that encourages shared structure across a category while maintaining instance details, and (3) using 3D background models to disentangle objects from the background. We show that 3D models of humans, cats, and dogs can be learned from 50-100 internet videos.