mlsanityData to 7 Oct 2026

Towards High-fidelity 3D Talking Avatar with Personalized Dynamic Texture

arXiv 2503.00495 ↗First posted 1 Mar 2025cs.CV cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Significant progress has been made for speech-driven 3D face animation, but most works focus on learning the motion of mesh/geometry, ignoring the impact of dynamic texture. In this work, we reveal that dynamic texture plays a key role in rendering high-fidelity talking avatars, and introduce a high-resolution 4D dataset TexTalk4D, consisting of 100 minutes of audio-synced scan-level meshes with detailed 8K dynamic textures from 100 subjects. Based on the dataset, we explore the inherent correlation between motion and texture, and propose a diffusion-based framework TexTalker to simultaneously generate facial motions and dynamic textures from speech. Furthermore, we propose a novel pivot-based style injection strategy to capture the complicity of different texture and motion styles, which allows disentangled control. TexTalker, as the first method to generate audio-synced facial motion with dynamic texture, not only outperforms the prior arts in synthesising facial motions, but also produces realistic textures that are consistent with the underlying facial movements. Project page: https://xuanchenli.github.io/TexTalk/.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 43 of its 75 references that are in the map. First posted on arXiv 1 Mar 2025.

Builds on

43 of 75 references.

References · 43

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  11. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  12. 2021Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Datacited by 961
31 more
  1. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  2. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  3. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  4. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
  5. 2022EMOCA: Emotion Driven Monocular Face Capture and Animationcited by 254
  6. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 208
  7. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  8. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  9. 2021Audio-Driven Emotional Video Portraitscited by 180
  10. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 141
  11. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  12. 2022Multiface: A Dataset for Neural Face Renderingcited by 87
  13. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Modelscited by 81
  14. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 77
  15. 2022Imitator: Personalized Speech-driven 3D Facial Animationcited by 73
  16. 2024DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generationcited by 61
  17. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55
  18. 2023RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatarscited by 45
  19. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  20. 2022DialogueNeRF: Towards Realistic Avatar Face-to-Face Conversation Video Generationcited by 30
  21. 2023FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learningcited by 29
  22. 2020Dynamic Facial Asset and Rig Generation from a Single Scancited by 21
  23. 2022Video-driven Neural Physically-based Facial Asset for Productioncited by 17
  24. 2023Speech-Driven 3D Face Animation with Composite and Regional Facial Movementscited by 17
  25. 2024MMHead: Towards Fine-grained Multi-modal 3D Facial Animationcited by 16
  26. 2021Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animationcited by 15
  27. 2023Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applicationscited by 15
  28. 2023MMFace4D: A Large-Scale Multi-Modal 4D Face Dataset for Audio-Driven 3D Face Animationcited by 8
  29. 2023CorrTalk: Correlation Between Hierarchical Speech and Facial Activity Variances for 3D Animationcited by 8
  30. 2024Topo4D: Topology-Preserving Gaussian Splatting for High-Fidelity 4D Head Capturecited by 8
  31. 2024AnimateMe: 4D Facial Expressions via Diffusion Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Controllable Video Generation: A Surveycited by 93
  2. 2025Audio-Driven Universal Gaussian Head Avatarscited by 6
  3. 2025Towards Seamless Interaction: Causal Turn-Level Modeling of Interactive 3D Conversational Head Dynamicscited by 3
  4. 2026Giving Faces Their Feelings Back: Explicit Emotion Control for Feedforward Single-Image 3D Head Avatarscited by 0
  5. 2026KM-Speaker: Keypoint-Based Style Control for High-Quality Speech-Driven 3D Facial Animation and Dialogue Localizationcited by 0