# Farm3D: Learning Articulated 3D Animals by Distilling 2D Diffusion

arXiv: https://arxiv.org/abs/2304.10535 · HTML: https://mlsanity.com/paper/2304.10535/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 34 of its 46 references that are in the map.
- First posted on arXiv 20 Apr 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 34)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [RealFusion: 360deg Reconstruction of Any Object from a Single Image](https://mlsanity.com/paper/2302.10663/) (2023)
- [What the DAAM: Interpreting Stable Diffusion Using Cross Attention](https://mlsanity.com/paper/2210.04885/) (2022)
- [Text-To-4D Dynamic Scene Generation](https://mlsanity.com/paper/2301.11280/) (2023)
- [BANMo: Building Animatable 3D Neural Models from Many Casual Videos](https://mlsanity.com/paper/2112.12761/) (2021)
- [Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild](https://mlsanity.com/paper/1911.11130/) (2019)
- [Fake it till you make it: Learning transferable representations from synthetic ImageNet clones](https://mlsanity.com/paper/2212.08420/) (2022)
- [Self-supervised Single-view 3D Reconstruction via Semantic Consistency](https://mlsanity.com/paper/2003.06473/) (2020)
- [LASR: Learning Articulated Shape Reconstruction from a Monocular Video](https://mlsanity.com/paper/2105.02976/) (2021)
- [VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models](https://mlsanity.com/paper/2211.11319/) (2022)
- [MagicPony: Learning Articulated 3D Animals in the Wild](https://mlsanity.com/paper/2211.12497/) (2022)
- [Articulation-aware Canonical Surface Mapping](https://mlsanity.com/paper/2004.00614/) (2020)
- [LASSIE: Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discovery](https://mlsanity.com/paper/2207.03434/) (2022)
- [DOVE: Learning Deformable 3D Objects by Watching Videos](https://mlsanity.com/paper/2107.10844/) (2021)

## Built on by (most cited first, 30 of 38)

- [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024)
- [Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors](https://mlsanity.com/paper/2306.17843/) (2023)
- [4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency](https://mlsanity.com/paper/2312.17225/) (2023)
- [GES: Generalized Exponential Splatting for Efficient Radiance Field Rendering](https://mlsanity.com/paper/2402.10128/) (2024)
- [Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstruction](https://mlsanity.com/paper/2504.07961/) (2025)
- [Learning the 3D Fauna of the Web](https://mlsanity.com/paper/2401.02400/) (2024)
- [DragAPart: Learning a Part-Level Motion Prior for Articulated Objects](https://mlsanity.com/paper/2403.15382/) (2024)
- [AnimatableDreamer: Text-Guided Non-rigid 3D Model Generation and Reconstruction with Canonical Score Distillation](https://mlsanity.com/paper/2312.03795/) (2023)
- [Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics](https://mlsanity.com/paper/2408.04631/) (2024)
- [DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundness](https://mlsanity.com/paper/2503.22677/) (2025)
- [Particulate: Feed-Forward 3D Object Articulation](https://mlsanity.com/paper/2512.11798/) (2025)
- [ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image Collections](https://mlsanity.com/paper/2306.04619/) (2023)
- [Ponymation: Learning Articulated 3D Animal Motions from Unlabeled Online Videos](https://mlsanity.com/paper/2312.13604/) (2023)
- [AniMer: Animal Pose and Shape Estimation Using Family Aware Transformer](https://mlsanity.com/paper/2412.00837/) (2024)
- [Survey on Modeling of Human-made Articulated Objects](https://mlsanity.com/paper/2403.14937/) (2024)
- [Articulated Kinematics Distillation from Video Diffusion Models](https://mlsanity.com/paper/2504.01204/) (2025)
- [DreaMo: Articulated 3D Reconstruction From A Single Casual Video](https://mlsanity.com/paper/2312.02617/) (2023)
- [GST: Precise 3D Human Body from a Single Image with Gaussian Splatting Transformers](https://mlsanity.com/paper/2409.04196/) (2024)
- [DualPM: Dual Posed-Canonical Point Maps for 3D Shape and Pose Reconstruction](https://mlsanity.com/paper/2412.04464/) (2024)
- [DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with Diffusion Priors](https://mlsanity.com/paper/2409.08278/) (2024)
- [Common3D: Self-Supervised Learning of 3D Morphable Models for Common Objects in Neural Feature Space](https://mlsanity.com/paper/2504.21749/) (2025)
- [Dessie: Disentanglement for Articulated 3D Horse Shape and Pose Estimation from Images](https://mlsanity.com/paper/2410.03438/) (2024)
- [AniMer+: Unified Pose and Shape Estimation Across Mammalia and Aves via Family-Aware Transformer](https://mlsanity.com/paper/2508.00298/) (2025)
- [GECO: Generative Image-to-3D within a SECOnd](https://mlsanity.com/paper/2405.20327/) (2024)
- [DIMO: Diverse 3D Motion Generation for Arbitrary Objects](https://mlsanity.com/paper/2511.07409/) (2025)
- [DreamBeast: Distilling 3D Fantastical Animals with Part-Aware Knowledge Transfer](https://mlsanity.com/paper/2409.08271/) (2024)
- [Instruct-Particulate: Scaling Feed-Forward 3D Object Articulation with Kinematic Control](https://mlsanity.com/paper/2606.14699/) (2026)
- [Towards Multi-Modal Animal Pose Estimation: A Survey and In-Depth Analysis](https://mlsanity.com/paper/2410.09312/) (2024)
- [4DEquine: Disentangling Motion and Appearance for 4D Equine Reconstruction from Monocular Video](https://mlsanity.com/paper/2603.10125/) (2026)
- [SAM 3D Animal: Promptable Animal 3D Reconstruction from Images in the Wild](https://mlsanity.com/paper/2605.07604/) (2026)

## Abstract (arXiv)

We present Farm3D, a method for learning category-specific 3D reconstructors for articulated objects, relying solely on "free" virtual supervision from a pre-trained 2D diffusion-based image generator. Recent approaches can learn a monocular network that predicts the 3D shape, albedo, illumination, and viewpoint of any object occurrence, given a collection of single-view images of an object category. However, these approaches heavily rely on manually curated clean training data, which are expensive to obtain. We propose a framework that uses an image generator, such as Stable Diffusion, to generate synthetic training data that are sufficiently clean and do not require further manual curation, enabling the learning of such a reconstruction network from scratch. Additionally, we incorporate the diffusion model as a score to enhance the learning process. The idea involves randomizing certain aspects of the reconstruction, such as viewpoint and illumination, generating virtual views of the reconstructed 3D object, and allowing the 2D network to assess the quality of the resulting image, thus providing feedback to the reconstructor. Unlike work based on distillation, which produces a single 3D asset for each textual prompt, our approach yields a monocular reconstruction network capable of outputting a controllable 3D asset from any given image, whether real or generated, in a single forward pass in a matter of seconds. Our network can be used for analysis, including monocular reconstruction, or for synthesis, generating articulated assets for real-time applications such as video games.