mlsanityData to 7 Oct 2026

Virtual Pets: Animatable Animal Generation in 3D Scenes

arXiv 2312.14154 ↗First posted 21 Dec 2023cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Toward unlocking the potential of generative models in immersive 4D experiences, we introduce Virtual Pet, a novel pipeline to model realistic and diverse motions for target animal species within a 3D environment. To circumvent the limited availability of 3D motion data aligned with environmental geometry, we leverage monocular internet videos and extract deformable NeRF representations for the foreground and static NeRF representations for the background. For this, we develop a reconstruction strategy, encompassing species-level shared template learning and per-video fine-tuning. Utilizing the reconstructed data, we then train a conditional 3D motion model to learn the trajectory and articulation of foreground animals in the context of 3D backgrounds. We showcase the efficacy of our pipeline with comprehensive qualitative and quantitative evaluations using cat videos. We also demonstrate versatility across unseen cats and indoor environments, producing temporally coherent 4D outputs for enriched virtual experiences.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 46 of its 69 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

46 of 69 references.

References · 46

  1. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  2. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  3. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  4. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  5. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  6. 2022Human Motion Diffusion Modelcited by 1,062
  7. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  8. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  9. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  10. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  11. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  12. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
34 more
  1. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  2. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  3. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  4. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  5. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  6. 2017On human motion prediction using recurrent neural networkscited by 336
  7. 2023TEXTure: Text-Guided Texturing of 3D Shapescited by 303
  8. 2019Resolving 3D Human Pose Ambiguities with 3D Scene Constraintscited by 294
  9. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 287
  10. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  11. 2015Structural-RNN: Deep Learning on Spatio-Temporal Graphscited by 234
  12. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Modelscited by 234
  13. 2022AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generationcited by 208
  14. 2023Text-To-4D Dynamic Scene Generationcited by 204
  15. 2021BANMo: Building Animatable 3D Neural Models from Many Casual Videoscited by 195
  16. 2017HP-GAN: Probabilistic 3D human motion prediction via GANcited by 176
  17. 2022Capturing and Inferring Dense Full-Body Human-Scene Contactcited by 176
  18. 2022Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-timecited by 164
  19. 2021Editable Free-viewpoint Video Using a Layered Neural Representationcited by 148
  20. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  21. 2019Generating 3D People in Scenes without Peoplecited by 127
  22. 2023Consistent4D: Consistent 360deg Dynamic Object Generation from Monocular Videocited by 124
  23. 2021LASR: Learning Articulated Shape Reconstruction from a Monocular Videocited by 115
  24. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  25. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 110
  26. 2023Affordance Diffusion: Synthesizing Hand-Object Interactionscited by 110
  27. 20223DILG: Irregular Latent Grids for 3D Generative Modelingcited by 92
  28. 2022Compositional Human-Scene Interaction Synthesis with Semantic Controlcited by 82
  29. 2018Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transitioncited by 78
  30. 2019Putting Humans in a Scene: Learning Affordance in 3D Indoor Environmentscited by 72
  31. 20204D Visualization of Dynamic Events from Unconstrained Multi-View Videoscited by 66
  32. 2021DOVE: Learning Deformable 3D Objects by Watching Videoscited by 61
  33. 2023Reconstructing Animatable Categories from Videoscited by 46
  34. 2023SceneTex: High-Quality Texture Synthesis for Indoor Scenes via Diffusion Priorscited by 43

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024GTR: Improving Large 3D Reconstruction Models through Geometry and Texture Refinementcited by 14
  2. 2025Advances and Trends in the 3D Reconstruction of the Shape and Motion of Animalscited by 2
  3. 2025Towards Affordance-Aware Articulation Synthesis for Rigged Objectscited by 0