# CVTHead: One-shot Controllable Head Avatar with Vertex-feature Transformer

arXiv: https://arxiv.org/abs/2311.06443 · HTML: https://mlsanity.com/paper/2311.06443/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 81 references that are in the map.
- First posted on arXiv 11 Nov 2023.

## Builds on (most cited first, 30 of 63)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization](https://mlsanity.com/paper/1905.05172/) (2019)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [Point-NeRF: Point-based Neural Radiance Fields](https://mlsanity.com/paper/2201.08845/) (2022)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://mlsanity.com/paper/1905.08233/) (2019)
- [StyleRig: Rigging StyleGAN for 3D Control over Portrait Images](https://mlsanity.com/paper/2004.00121/) (2020)
- [Mesh Graphormer](https://mlsanity.com/paper/2104.00272/) (2021)
- [ARCH: Animatable Reconstruction of Clothed Humans](https://mlsanity.com/paper/2004.04572/) (2020)

## Built on by (most cited first, 24 of 24)

- [Generalizable and Animatable Gaussian Head Avatar](https://mlsanity.com/paper/2410.07971/) (2024)
- [HeadGAP: Few-Shot 3D Head Avatar via Generalizable Gaussian Priors](https://mlsanity.com/paper/2408.06019/) (2024)
- [LAM: Large Avatar Model for One-shot Animatable Gaussian Head](https://mlsanity.com/paper/2502.17796/) (2025)
- [MegActor: Harness the Power of Raw Video for Vivid Portrait Animation](https://mlsanity.com/paper/2405.20851/) (2024)
- [FixTalk: Taming Identity Leakage for High-Quality Talking Head Generation in Extreme Cases](https://mlsanity.com/paper/2507.01390/) (2025)
- [MegActor-Sigma: Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion Transformer](https://mlsanity.com/paper/2408.14975/) (2024)
- [FastAvatar: Instant 3D Gaussian Splatting for Faces from Single Unconstrained Poses](https://mlsanity.com/paper/2508.18389/) (2025)
- [FastAvatar: Towards Unified and Fast 3D Avatar Reconstruction with Large Gaussian Reconstruction Transformers](https://mlsanity.com/paper/2508.19754/) (2025)
- [Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functions](https://mlsanity.com/paper/2507.02900/) (2025)
- [Identity-Preserving Video Dubbing Using Motion Warping](https://mlsanity.com/paper/2501.04586/) (2025)
- [Dual Audio-Centric Modality Coupling for Talking Head Generation](https://mlsanity.com/paper/2503.22728/) (2025)
- [Unmasking Puppeteers: Leveraging Biometric Leakage to Expose Impersonation in AI-Based Videoconferencing](https://mlsanity.com/paper/2510.03548/) (2025)
- [Mask-Free Audio-driven Talking Face Generation for Enhanced Visual Quality and Identity Preservation](https://mlsanity.com/paper/2507.20953/) (2025)
- [Instruction-Driven 3D Facial Expression Generation and Transition](https://mlsanity.com/paper/2601.08179/) (2026)
- [OMG-Avatar: One-shot Multi-LOD Gaussian Head Avatar](https://mlsanity.com/paper/2603.01506/) (2026)
- [Eye-See-You: Reverse Pass-Through VR and Head Avatars](https://mlsanity.com/paper/2505.18869/) (2025)
- [LiftAvatar: Kinematic-Space Completion for Expression-Controlled 3D Gaussian Avatar Animation](https://mlsanity.com/paper/2603.02129/) (2026)
- [GenFusion: Feed-forward Human Performance Capture via Progressive Canonical Space Updates](https://mlsanity.com/paper/2603.28997/) (2026)
- [SplitAvatar: One-shot Head Avatar with Autoregressive Gaussian Splitting](https://mlsanity.com/paper/2605.25751/) (2026)
- [Splatshot: 3D Face Avatar Generation from a Single Unconstrained Photo](https://mlsanity.com/paper/2606.01493/) (2026)
- [SpatialAvatar-0: High-Quality 4D Head Avatar with Multi-Stage Reconstruction](https://mlsanity.com/paper/2606.15659/) (2026)
- [FFAvatar: Feed-Forward 4D Head Avatar Reconstruction from Sparse Portrait Images](https://mlsanity.com/paper/2606.30347/) (2026)
- [Learning Semantic Inpainting for Animatable Gaussian Head Avatars](https://mlsanity.com/paper/2609.38343/) (2026)
- [GLARE: Generating Listening Heads with Appropriate Reactions](https://mlsanity.com/paper/2609.40317/) (2026)

## Abstract (arXiv)

Reconstructing personalized animatable head avatars has significant implications in the fields of AR/VR. Existing methods for achieving explicit face control of 3D Morphable Models (3DMM) typically rely on multi-view images or videos of a single subject, making the reconstruction process complex. Additionally, the traditional rendering pipeline is time-consuming, limiting real-time animation possibilities. In this paper, we introduce CVTHead, a novel approach that generates controllable neural head avatars from a single reference image using point-based neural rendering. CVTHead considers the sparse vertices of mesh as the point set and employs the proposed Vertex-feature Transformer to learn local feature descriptors for each vertex. This enables the modeling of long-range dependencies among all the vertices. Experimental results on the VoxCeleb dataset demonstrate that CVTHead achieves comparable performance to state-of-the-art graphics-based methods. Moreover, it enables efficient rendering of novel human heads with various expressions, head poses, and camera views. These attributes can be explicitly controlled using the coefficients of 3DMMs, facilitating versatile and realistic animation in real-time scenarios.