# Bring Your Own Character: A Holistic Solution for Automatic Facial Animation Generation of Customized Characters

arXiv: https://arxiv.org/abs/2402.13724 · HTML: https://mlsanity.com/paper/2402.13724/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 38 references that are in the map.
- First posted on arXiv 21 Feb 2024.

## Builds on (most cited first, 8 of 8)

- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network](https://mlsanity.com/paper/1803.07835/) (2018)
- [End-to-end 3D face reconstruction with deep neural networks](https://mlsanity.com/paper/1704.05020/) (2017)
- [Personalized Face Modeling for Improved Face Reconstruction and Motion Retargeting](https://mlsanity.com/paper/2007.06759/) (2020)
- [Fast and Robust Face-to-Parameter Translation for Game Character Auto-Creation](https://mlsanity.com/paper/2008.07132/) (2020)
- [Face-to-Parameter Translation for Game Character Auto-Creation](https://mlsanity.com/paper/1909.01064/) (2019)
- [Unsupervised Creation of Parameterized Avatars](https://mlsanity.com/paper/1704.05693/) (2017)

## Built on by (most cited first, 6 of 6)

- [MixedGaussianAvatar: Realistically and Geometrically Accurate Head Avatar via Mixed 2D-3D Gaussians](https://mlsanity.com/paper/2412.04955/) (2024)
- [DiffusionTalker: Efficient and Compact Speech-Driven 3D Talking Head via Personalizer-Guided Distillation](https://mlsanity.com/paper/2503.18159/) (2025)
- [RSATalker: Realistic Socially-Aware Talking Head Generation for Multi-Turn Conversation](https://mlsanity.com/paper/2601.10606/) (2026)
- [FactorPortrait: Controllable Portrait Animation via Disentangled Expression, Pose, and Viewpoint](https://mlsanity.com/paper/2512.11645/) (2025)
- [Face-MakeUpV2: Facial Consistency Learning for Controllable Text-to-Image Generation](https://mlsanity.com/paper/2510.21775/) (2025)
- [Conversational Human Audio-visual Talking Dialogue Generation](https://mlsanity.com/paper/2607.02799/) (2026)

## Abstract (arXiv)

Animating virtual characters has always been a fundamental research problem in virtual reality (VR). Facial animations play a crucial role as they effectively convey emotions and attitudes of virtual humans. However, creating such facial animations can be challenging, as current methods often involve utilization of expensive motion capture devices or significant investments of time and effort from human animators in tuning animation parameters. In this paper, we propose a holistic solution to automatically animate virtual human faces. In our solution, a deep learning model was first trained to retarget the facial expression from input face images to virtual human faces by estimating the blendshape coefficients. This method offers the flexibility of generating animations with characters of different appearances and blendshape topologies. Second, a practical toolkit was developed using Unity 3D, making it compatible with the most popular VR applications. The toolkit accepts both image and video as input to animate the target virtual human faces and enables users to manipulate the animation results. Furthermore, inspired by the spirit of Human-in-the-loop (HITL), we leveraged user feedback to further improve the performance of the model and toolkit, thereby increasing the customization properties to suit user preferences. The whole solution, for which we will make the code public, has the potential to accelerate the generation of facial animations for use in VR applications.