mlsanityData to 7 Oct 2026

Unsupervised Volumetric Animation

arXiv 2301.11326 ↗First posted 26 Jan 2023cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

We propose a novel approach for unsupervised 3D animation of non-rigid deformable objects. Our method learns the 3D structure and dynamics of objects solely from single-view RGB videos, and can decompose them into semantically meaningful parts that can be tracked and animated. Using a 3D autodecoder framework, paired with a keypoint estimator via a differentiable PnP algorithm, our model learns the underlying object geometry and parts decomposition in an entirely unsupervised manner. This allows it to perform 3D segmentation, 3D keypoint estimation, novel view synthesis, and animation. We primarily evaluate the framework on two video datasets: VoxCeleb 2562 and TEDXPeople 2562. In addition, on the Cats 2562 image dataset, we show it even learns compelling 3D geometry from still images. Finally, we show our model can obtain animatable 3D objects from a single or few images. Code and visual results available on our project website, see https://snap-research.github.io/unsupervised-volumetric-animation .

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 70 of its 82 references that are in the map. First posted on arXiv 26 Jan 2023.

Builds on

70 of 82 references.

References · 70

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  6. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  7. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  8. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  9. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  10. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  11. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  12. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
38 more
  1. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  2. 2020First Order Motion Model for Image Animationcited by 698
  3. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  4. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 569
  5. 2022HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Videocited by 483
  6. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  7. 2020Learning an Animatable Detailed 3D Face Model from In-The-Wild Imagescited by 475
  8. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
  9. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  10. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  11. 2021Animatable Neural Radiance Fields for Modeling Dynamic Human Bodiescited by 361
  12. 2017Learning from Synthetic Humanscited by 358
  13. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  14. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  15. 2018Everybody Dance Nowcited by 313
  16. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  17. 2021StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generationcited by 289
  18. 2019Few-Shot Adversarial Learning of Realistic Neural Talking Head Modelscited by 284
  19. 2021Motion Representations for Articulated Animationcited by 253
  20. 2022NeuMan: Neural Human Radiance Field from a Single Videocited by 244
  21. 2019Kornia: an Open Source Differentiable Computer Vision Library for PyTorchcited by 231
  22. 2021GRAM: Generative Radiance Manifolds for 3D-Aware Image Generationcited by 216
  23. 2022Thin-Plate Spline Motion Model for Image Animationcited by 209
  24. 2018Animating Arbitrary Objects via Deep Motion Transfercited by 206
  25. 2019Few-shot Video-to-Video Synthesiscited by 196
  26. 2021BANMo: Building Animatable 3D Neural Models from Many Casual Videoscited by 195
  27. 2018X2Face: A network for controlling face generation by using images, audio, and pose codescited by 189
  28. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  29. 2019Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wildcited by 180
  30. 2021InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Renderingcited by 180
  31. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 172
  32. 2022TAVA: Template-free Animatable Volumetric Actorscited by 172
  33. 2020BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Imagescited by 166
  34. 2019Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesiscited by 161
  35. 2017Optimizing the Latent Space of Generative Networkscited by 157
  36. 2022MegaPortraits: One-shot Megapixel Neural Head Avatarscited by 131
  37. 2020Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatarscited by 116
  38. 2021LASR: Learning Articulated Shape Reconstruction from a Monocular Videocited by 115

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 3; 2025: 2; 2026: 3.

18 papers.

Sort:

By area:

  1. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  2. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  3. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Modelscited by 234
  4. 2023AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Controlcited by 84
  5. 2024EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatarscited by 65
  6. 2023Portrait4D: Learning One-Shot 4D Head Avatar Synthesis using Synthetic Datacited by 55
  7. 2023SceneWiz3D: Towards Text-guided 3D Scene Compositioncited by 54
  8. 2023AutoDecoding Latent 3D Diffusion Modelscited by 50
  9. 2023SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priorscited by 43
  10. 2023Implicit Neural Head Synthesis via Controllable Local Deformation Fieldscited by 14
  11. 2024Correspondence-Free Non-Rigid Point Set Registration Using Unsupervised Clustering Analysiscited by 10
  12. 20233D-Aware Talking-Head Video Motion Transfercited by 4
6 more
  1. 2025Occlusion-aware Non-Rigid Point Cloud Registration via Unsupervised Neural Deformation Correntropycited by 4
  2. 2024NToP: NeRF-Powered Large-scale Dataset Generation for 2D and 3D Human Pose Estimation in Top-View Fisheye Imagescited by 2
  3. 2026CustomTex: High-fidelity Indoor Scene Texturing via Multi-Reference Customizationcited by 1
  4. 2025Instant Expressive Gaussian Head Avatars at Over 100 FPScited by 0
  5. 2026High-Fidelity 3D Facial Avatar Synthesis with Controllable Fine-Grained Expressionscited by 0
  6. 2026SpatialAvatar-0: High-Quality 4D Head Avatar with Multi-Stage Reconstructioncited by 0