# Expressive Telepresence via Modular Codec Avatars

arXiv: https://arxiv.org/abs/2008.11789 · HTML: https://mlsanity.com/paper/2008.11789/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Aug 2020.

## Built on by (most cited first, 17 of 17)

- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [Pixel Codec Avatars](https://mlsanity.com/paper/2104.04638/) (2021)
- [Multiface: A Dataset for Neural Face Rendering](https://mlsanity.com/paper/2207.11243/) (2022)
- [AvatarReX: Real-time Expressive Full-body Avatars](https://mlsanity.com/paper/2305.04789/) (2023)
- [LatentAvatar: Learning Latent Expression Code for Expressive Neural Head Avatar](https://mlsanity.com/paper/2305.01190/) (2023)
- [XAGen: 3D Expressive Human Avatars Generation](https://mlsanity.com/paper/2311.13574/) (2023)
- [Quark: Real-time, High-resolution, and General Neural View Synthesis](https://mlsanity.com/paper/2411.16680/) (2024)
- [Robust Egocentric Photo-realistic Facial Expression Transfer for Virtual Reality](https://mlsanity.com/paper/2104.04794/) (2021)
- [Disentangled Clothed Avatar Generation with Layered Representation](https://mlsanity.com/paper/2501.04631/) (2025)
- [LUCAS: Layered Universal Codec Avatars](https://mlsanity.com/paper/2502.19739/) (2025)
- [Force-Aware Interface via Electromyography for Natural VR/AR Interaction](https://mlsanity.com/paper/2210.01225/) (2022)
- [Auto-CARD: Efficient and Robust Codec Avatar Driving for Real-time Mobile Telepresence](https://mlsanity.com/paper/2304.11835/) (2023)
- [High-Fidelity 3D Head Avatars Reconstruction through Spatially-Varying Expression Conditioned Neural Radiance Field](https://mlsanity.com/paper/2310.06275/) (2023)
- [Unlocking Visual Secrets: Inverting Features with Diffusion Priors for Image Reconstruction](https://mlsanity.com/paper/2412.10448/) (2024)
- [F-CAD: A Framework to Explore Hardware Accelerators for Codec Avatar Decoding](https://mlsanity.com/paper/2103.04958/) (2021)
- [Unmasking Puppeteers: Leveraging Biometric Leakage to Expose Impersonation in AI-Based Videoconferencing](https://mlsanity.com/paper/2510.03548/) (2025)
- [LiP-Flow: Learning Inference-time Priors for Codec Avatars via Normalizing Flows in Latent Space](https://mlsanity.com/paper/2203.07881/) (2022)

## Abstract (arXiv)

VR telepresence consists of interacting with another human in a virtual space represented by an avatar. Today most avatars are cartoon-like, but soon the technology will allow video-realistic ones. This paper aims in this direction and presents Modular Codec Avatars (MCA), a method to generate hyper-realistic faces driven by the cameras in the VR headset. MCA extends traditional Codec Avatars (CA) by replacing the holistic models with a learned modular representation. It is important to note that traditional person-specific CAs are learned from few training samples, and typically lack robustness as well as limited expressiveness when transferring facial expressions. MCAs solve these issues by learning a modulated adaptive blending of different facial components as well as an exemplar-based latent alignment. We demonstrate that MCA achieves improved expressiveness and robustness w.r.t to CA in a variety of real-world datasets and practical scenarios. Finally, we showcase new applications in VR telepresence enabled by the proposed model.