# OT-Talk: Animating 3D Talking Head with Optimal Transportation arXiv: https://arxiv.org/abs/2505.01932 ยท HTML: https://mlsanity.com/paper/2505.01932/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 53 references that are in the map. - First posted on arXiv 3 May 2025. ## Builds on (most cited first, 28 of 28) - [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020) - [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013) - [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016) - [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018) - [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020) - [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014) - [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021) - [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019) - [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018) - [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021) - [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021) - [Audio-Driven Emotional Video Portraits](https://mlsanity.com/paper/2104.07452/) (2021) - [Realistic Speech-Driven Facial Animation with GANs](https://mlsanity.com/paper/1906.06337/) (2019) - [EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation](https://mlsanity.com/paper/2303.11089/) (2023) - [Talking-head Generation with Rhythmic Head Motion](https://mlsanity.com/paper/2007.08547/) (2020) - [Learning Speech-driven 3D Conversational Gestures from Video](https://mlsanity.com/paper/2102.06837/) (2021) - [Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation](https://mlsanity.com/paper/1905.02876/) (2019) - [Multiface: A Dataset for Neural Face Rendering](https://mlsanity.com/paper/2207.11243/) (2022) - [DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models](https://mlsanity.com/paper/2310.00434/) (2023) - [FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusion](https://mlsanity.com/paper/2309.11306/) (2023) - [SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Faces](https://mlsanity.com/paper/2306.10799/) (2023) - [VisemeNet: Audio-Driven Animator-Centric Speech Animation](https://mlsanity.com/paper/1805.09488/) (2018) - [Deformable Shape Completion with Graph Convolutional Autoencoders](https://mlsanity.com/paper/1712.00268/) (2017) - [MeshGAN: Non-linear 3D Morphable Models of Faces](https://mlsanity.com/paper/1903.10384/) (2019) - [The varifold representation of non-oriented shapes for diffeomorphic registration](https://mlsanity.com/paper/1304.6108/) (2013) - [Geometry-guided Dense Perspective Network for Speech-Driven Facial Animation](https://mlsanity.com/paper/2008.10004/) (2020) - [Speech-Driven 3D Face Animation with Composite and Regional Facial Movements](https://mlsanity.com/paper/2308.05428/) (2023) - [Talking Head from Speech Audio using a Pre-trained Image Generator](https://mlsanity.com/paper/2209.04252/) (2022) ## Built on by (most cited first, 6 of 6) - [Neural Geometry Image-Based Representations with Optimal Transport (OT)](https://mlsanity.com/paper/2511.18679/) (2025) - [Inverse Rendering for High-Genus Surface Meshes from Multi-View Images](https://mlsanity.com/paper/2511.18680/) (2025) - [Learning Domain Agnostic Latent Embeddings of 3D Faces for Zero-shot Animal Expression Transfer](https://mlsanity.com/paper/2601.06484/) (2026) - [Inverse Rendering for High-Genus 3D Surface Meshes from Multi-view Images with Persistent Homology Priors](https://mlsanity.com/paper/2601.12155/) (2026) - [3DXTalker: Unifying Identity, Lip Sync, Emotion, and Spatial Dynamics in Expressive 3D Talking Avatars](https://mlsanity.com/paper/2602.10516/) (2026) - [CapTalk: Text-Guided Stylization and Speech-Driven 3D Head Animation](https://mlsanity.com/paper/2605.29316/) (2026) ## Abstract (arXiv) Animating 3D head meshes using audio inputs has significant applications in AR/VR, gaming, and entertainment through 3D avatars. However, bridging the modality gap between speech signals and facial dynamics remains a challenge, often resulting in incorrect lip syncing and unnatural facial movements. To address this, we propose OT-Talk, the first approach to leverage optimal transportation to optimize the learning model in talking head animation. Building on existing learning frameworks, we utilize a pre-trained Hubert model to extract audio features and a transformer model to process temporal sequences. Unlike previous methods that focus solely on vertex coordinates or displacements, we introduce Chebyshev Graph Convolution to extract geometric features from triangulated meshes. To measure mesh dissimilarities, we go beyond traditional mesh reconstruction errors and velocity differences between adjacent frames. Instead, we represent meshes as probability measures and approximate their surfaces. This allows us to leverage the sliced Wasserstein distance for modeling mesh variations. This approach facilitates the learning of smooth and accurate facial motions, resulting in coherent and natural facial animations. Our experiments on two public audio-mesh datasets demonstrate that our method outperforms state-of-the-art techniques both quantitatively and qualitatively in terms of mesh reconstruction accuracy and temporal alignment. In addition, we conducted a user perception study with 20 volunteers to further assess the effectiveness of our approach.