# TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humans

arXiv: https://arxiv.org/abs/2409.16666 · HTML: https://mlsanity.com/paper/2409.16666/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 68 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Builds on (most cited first, 30 of 58)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans](https://mlsanity.com/paper/2012.15838/) (2020)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video](https://mlsanity.com/paper/2201.04127/) (2022)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)
- [AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis](https://mlsanity.com/paper/2103.11078/) (2021)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians](https://mlsanity.com/paper/2312.02069/) (2023)
- [NeuMan: Neural Human Radiance Field from a Single Video](https://mlsanity.com/paper/2203.12575/) (2022)
- [A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose](https://mlsanity.com/paper/2102.06199/) (2021)
- [HeadNeRF: A Real-time NeRF-based Parametric Head Model](https://mlsanity.com/paper/2112.05637/) (2021)
- [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://mlsanity.com/paper/2009.09960/) (2020)
- [Neural Articulated Radiance Field](https://mlsanity.com/paper/2104.03110/) (2021)
- [3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting](https://mlsanity.com/paper/2312.09228/) (2023)

## Built on by (most cited first, 5 of 5)

- [A Unit Enhancement and Guidance Framework for Audio-Driven Avatar Video Generation](https://mlsanity.com/paper/2505.03603/) (2025)
- [Mask-Free Audio-driven Talking Face Generation for Enhanced Visual Quality and Identity Preservation](https://mlsanity.com/paper/2507.20953/) (2025)
- [VRGaussianAvatar: Integrating 3D Gaussian Avatars into VR](https://mlsanity.com/paper/2602.01674/) (2026)
- [Text-Driven Emotionally Continuous Talking Face Generation](https://mlsanity.com/paper/2603.06071/) (2026)
- [PAGE: Parts-Aware Guidance for Co-Speech Gesture Portrait Video Generation](https://mlsanity.com/paper/2610.13301/) (2026)

## Abstract (arXiv)

We introduce a novel framework that learns a dynamic neural radiance field (NeRF) for full-body talking humans from monocular videos. Prior work represents only the body pose or the face. However, humans communicate with their full body, combining body pose, hand gestures, as well as facial expressions. In this work, we propose TalkinNeRF, a unified NeRF-based network that represents the holistic 4D human motion. Given a monocular video of a subject, we learn corresponding modules for the body, face, and hands, that are combined together to generate the final result. To capture complex finger articulation, we learn an additional deformation field for the hands. Our multi-identity representation enables simultaneous training for multiple subjects, as well as robust animation under completely unseen poses. It can also generalize to novel identities, given only a short video as input. We demonstrate state-of-the-art performance for animating full-body talking humans, with fine-grained hand articulation and facial expressions.