# Animatable Virtual Humans: Learning pose-dependent human representations in UV space for interactive performance synthesis

arXiv: https://arxiv.org/abs/2310.03615 · HTML: https://mlsanity.com/paper/2310.03615/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose](https://mlsanity.com/paper/2102.06199/) (2021)
- [Neural Head Avatars from Monocular RGB Videos](https://mlsanity.com/paper/2112.01554/) (2021)
- [Tex2Shape: Detailed Full Human Body Geometry From a Single Image](https://mlsanity.com/paper/1904.08645/) (2019)
- [InstantAvatar: Learning Avatars from Monocular Video in 60 Seconds](https://mlsanity.com/paper/2212.10550/) (2022)
- [Collaborative Regression of Expressive Bodies using Moderation](https://mlsanity.com/paper/2105.05301/) (2021)
- [Pixel Codec Avatars](https://mlsanity.com/paper/2104.04638/) (2021)
- [Structured Local Radiance Fields for Human Avatar Modeling](https://mlsanity.com/paper/2203.14478/) (2022)
- [AvatarReX: Real-time Expressive Full-body Avatars](https://mlsanity.com/paper/2305.04789/) (2023)
- [SMPLpix: Neural Avatars from 3D Human Models](https://mlsanity.com/paper/2008.06872/) (2020)
- [X-Avatar: Expressive Human Avatars](https://mlsanity.com/paper/2303.04805/) (2023)
- [HumanGAN: A Generative Model of Humans Images](https://mlsanity.com/paper/2103.06902/) (2021)
- [Surface-Aligned Neural Radiance Fields for Controllable 3D Human Synthesis](https://mlsanity.com/paper/2201.01683/) (2022)
- [Generalizable Neural Radiance Fields for Novel View Synthesis with Transformer](https://mlsanity.com/paper/2206.05375/) (2022)
- [DEMEA: Deep Mesh Autoencoders for Non-Rigidly Deforming Objects](https://mlsanity.com/paper/1905.10290/) (2019)
- [Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation](https://mlsanity.com/paper/2001.04947/) (2020)
- [Imposing Temporal Consistency on Deep Monocular Body Shape and Pose Estimation](https://mlsanity.com/paper/2202.03074/) (2022)

## Built on by (most cited first, 3 of 3)

- [ExFMan: Rendering 3D Dynamic Humans with Hybrid Monocular Blurry Frames and Events](https://mlsanity.com/paper/2409.14103/) (2024)
- [EPSilon: Efficient Point Sampling for Lightening of Hybrid-based 3D Avatar Generation](https://mlsanity.com/paper/2507.13648/) (2025)
- [Event-guided 3D Gaussian Splatting for Dynamic Human and Scene Reconstruction](https://mlsanity.com/paper/2509.18566/) (2025)

## Abstract (arXiv)

We propose a novel representation of virtual humans for highly realistic real-time animation and rendering in 3D applications. We learn pose dependent appearance and geometry from highly accurate dynamic mesh sequences obtained from state-of-the-art multiview-video reconstruction. Learning pose-dependent appearance and geometry from mesh sequences poses significant challenges, as it requires the network to learn the intricate shape and articulated motion of a human body. However, statistical body models like SMPL provide valuable a-priori knowledge which we leverage in order to constrain the dimension of the search space enabling more efficient and targeted learning and define pose-dependency. Instead of directly learning absolute pose-dependent geometry, we learn the difference between the observed geometry and the fitted SMPL model. This allows us to encode both pose-dependent appearance and geometry in the consistent UV space of the SMPL model. This approach not only ensures a high level of realism but also facilitates streamlined processing and rendering of virtual humans in real-time scenarios.