# CASA: Category-agnostic Skeletal Animal Reconstruction

arXiv: https://arxiv.org/abs/2211.03568 · HTML: https://mlsanity.com/paper/2211.03568/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 90 references that are in the map.
- First posted on arXiv 4 Nov 2022.

## Builds on (most cited first, 30 of 57)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization](https://mlsanity.com/paper/2004.00452/) (2020)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields](https://mlsanity.com/paper/2112.05139/) (2021)
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [3D Menagerie: Modeling the 3D shape and pose of animals](https://mlsanity.com/paper/1611.07700/) (2016)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [BANMo: Building Animatable 3D Neural Models from Many Casual Videos](https://mlsanity.com/paper/2112.12761/) (2021)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)

## Built on by (most cited first, 30 of 35)

- [Tracking Everything Everywhere All at Once](https://mlsanity.com/paper/2306.05422/) (2023)
- [Physical Property Understanding from Language-Embedded Feature Fields](https://mlsanity.com/paper/2404.04242/) (2024)
- [Reconstructing Animatable Categories from Videos](https://mlsanity.com/paper/2305.06351/) (2023)
- [Building Rearticulable Models for Arbitrary 3D Objects from 4D Point Clouds](https://mlsanity.com/paper/2306.00979/) (2023)
- [Uni4D: Unifying Visual Foundation Models for 4D Modeling from a Single Video](https://mlsanity.com/paper/2503.21761/) (2025)
- [Anymate: A Dataset and Baselines for Learning 3D Object Rigging](https://mlsanity.com/paper/2505.06227/) (2025)
- [DRiVE: Diffusion-based Rigging Empowers Generation of Versatile and Expressive Characters](https://mlsanity.com/paper/2411.17423/) (2024)
- [DepR: Depth Guided Single-view Scene Reconstruction with Instance-level Diffusion](https://mlsanity.com/paper/2507.22825/) (2025)
- [Ponymation: Learning Articulated 3D Animal Motions from Unlabeled Online Videos](https://mlsanity.com/paper/2312.13604/) (2023)
- [Bayesian Diffusion Models for 3D Shape Reconstruction](https://mlsanity.com/paper/2403.06973/) (2024)
- [Template-free Articulated Neural Point Clouds for Reposable View Synthesis](https://mlsanity.com/paper/2305.19065/) (2023)
- [Survey on Modeling of Human-made Articulated Objects](https://mlsanity.com/paper/2403.14937/) (2024)
- [RigGS: Rigging of 3D Gaussians for Modeling Articulated Objects in Videos](https://mlsanity.com/paper/2503.16822/) (2025)
- [Template-free Articulated Gaussian Splatting for Real-time Reposable Dynamic View Synthesis](https://mlsanity.com/paper/2412.05570/) (2024)
- [DreaMo: Articulated 3D Reconstruction From A Single Casual Video](https://mlsanity.com/paper/2312.02617/) (2023)
- [SRIF: Semantic Shape Registration Empowered by Diffusion-based Image Morphing and Flow Estimation](https://mlsanity.com/paper/2409.11682/) (2024)
- [Generative Zoo](https://mlsanity.com/paper/2412.08101/) (2024)
- [CAMM: Building Category-Agnostic and Animatable 3D Models from Monocular Videos](https://mlsanity.com/paper/2304.06937/) (2023)
- [MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos](https://mlsanity.com/paper/2512.10881/) (2025)
- [3D Human Keypoints Estimation From Point Clouds in the Wild Without Human Labels](https://mlsanity.com/paper/2306.04745/) (2023)
- [Learning Implicit Representation for Reconstructing Articulated Objects](https://mlsanity.com/paper/2401.08809/) (2024)
- [Reconstructing Animals and the Wild](https://mlsanity.com/paper/2411.18807/) (2024)
- [One Video, One World: Turning Monocular Video into Physical 4D Scenes](https://mlsanity.com/paper/2606.31388/) (2026)
- [TextANIMAR: Text-based 3D Animal Fine-Grained Retrieval](https://mlsanity.com/paper/2304.06053/) (2023)
- [ARMO: Autoregressive Rigging for Multi-Category Objects](https://mlsanity.com/paper/2503.20663/) (2025)
- [DogMo: A Large-Scale Multi-View RGB-D Dataset for 4D Canine Motion Recovery](https://mlsanity.com/paper/2510.24117/) (2025)
- [NeuralFur: Animal Fur Reconstruction From Multi-View Images](https://mlsanity.com/paper/2601.12481/) (2026)
- [SketchANIMAR: Sketch-based 3D Animal Fine-Grained Retrieval](https://mlsanity.com/paper/2304.05731/) (2023)
- [Advances and Trends in the 3D Reconstruction of the Shape and Motion of Animals](https://mlsanity.com/paper/2508.16062/) (2025)
- [MimiCAT: Mimic with Correspondence-Aware Cascade-Transformer for Category-Free 3D Pose Transfer](https://mlsanity.com/paper/2511.18370/) (2025)

## Abstract (arXiv)

Recovering the skeletal shape of an animal from a monocular video is a longstanding challenge. Prevailing animal reconstruction methods often adopt a control-point driven animation model and optimize bone transforms individually without considering skeletal topology, yielding unsatisfactory shape and articulation. In contrast, humans can easily infer the articulation structure of an unknown animal by associating it with a seen articulated character in their memory. Inspired by this fact, we present CASA, a novel Category-Agnostic Skeletal Animal reconstruction method consisting of two major components: a video-to-shape retrieval process and a neural inverse graphics framework. During inference, CASA first retrieves an articulated shape from a 3D character assets bank so that the input video scores highly with the rendered image, according to a pretrained language-vision model. CASA then integrates the retrieved character into an inverse graphics framework and jointly infers the shape deformation, skeleton structure, and skinning weights through optimization. Experiments validate the efficacy of CASA regarding shape reconstruction and articulation. We further demonstrate that the resulting skeletal-animated characters can be used for re-animation.