mlsanityData to 14 Oct 2026

ARTalk: Speech-Driven 3D Head Animation via Autoregressive Model

arXiv 2502.20323 ↗First posted 27 Feb 2025cs.CVcited by 29

Not on the map. Why?

Abstract (arXiv)

Speech-driven 3D facial animation aims to generate realistic lip movements and facial expressions for 3D head models from arbitrary audio clips. Although existing diffusion-based methods are capable of producing natural motions, their slow generation speed limits their application potential. In this paper, we introduce a novel autoregressive model that achieves real-time generation of highly synchronized lip movements and realistic head poses and eye blinks by learning a mapping from speech to a multi-scale motion codebook. Furthermore, our model can adapt to unseen speaking styles, enabling the creation of 3D talking avatars with unique personal styles beyond the identities seen during training. Extensive evaluations and user studies demonstrate that our method outperforms existing approaches in lip synchronization accuracy and perceived quality.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 38 of its 55 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

38 of 55 references.

References · 38

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
  7. 2022Human Motion Diffusion Modelcited by 1,080
  8. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  9. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  10. 2020Learning an Animatable Detailed 3D Face Model from In-The-Wild Imagescited by 477
  11. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  12. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 259
26 more
  1. 2022EMOCA: Emotion Driven Monocular Face Capture and Animationcited by 255
  2. 2022Towards Robust Blind Face Restoration with Codebook Lookup Transformercited by 231
  3. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 190
  4. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 190
  5. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 184
  6. 2022VFHQ: A High-Quality Dataset and Benchmark for Video Face Super-Resolutioncited by 176
  7. 2022Towards Metrical Reconstruction of Human Facescited by 166
  8. 2020DiffusionNet: Discretization Agnostic Learning on Surfacescited by 164
  9. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 142
  10. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 129
  11. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  12. 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
  13. 2024Generalizable and Animatable Gaussian Head Avatarcited by 97
  14. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Modelscited by 81
  15. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 78
  16. 2024GPAvatar: Generalizable and Precise Head Avatar from Image(s)cited by 74
  17. 2023SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Facescited by 68
  18. 2024Portrait4D-v2: Pseudo Multi-View Data Creates Better 4D Head Synthesizercited by 64
  19. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55
  20. 2024MultiTalk: Enhancing 3D Talking Head Generation Across Languages with Multilingual Video Datasetcited by 28
  21. 2024Say Anything with Any Stylecited by 26
  22. 2024ScanTalk: 3D Talking Heads from Unregistered Scanscited by 24
  23. 2024DiffSpeaker: Speech-Driven 3D Facial Animation with Diffusion Transformercited by 18
  24. 2024Learn2Talk: 3D Talking Face Learns from 2D Talking Facecited by 16
  25. 2024MMHead: Towards Fine-grained Multi-modal 3D Facial Animationcited by 16
  26. 2023Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applicationscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 23.

29 papers.

Sort:

By area:

  1. 2025UniLS: End-to-End Audio-Driven Avatars for Unified Listening and Speakingcited by 11
  2. 2025Towards Interactive Intelligence for Digital Humanscited by 10
  3. 2025ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Bodycited by 6
  4. 2025REST: Diffusion-based Real-time End-to-end Streaming Talking Head Generation via ID-Context Caching and Asynchronous Streaming Distillationcited by 4
  5. 2025FluentAvatar: Flicker-Free Talking-Head Animation via Phoneme-Guided Autoregressive Modelingcited by 1
  6. 2026MANGO:Natural Multi-speaker 3D Talking Head Generation via 2D-Lifted Enhancementcited by 1
  7. 2026A2-LLM: An End-to-end Conversational Audio Avatar Large Language Modelcited by 1
  8. 2026EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agentscited by 1
  9. 2026FMReward: Aligning and Evaluating Audio-Driven 3D Facial Animation with Human Preferencescited by 1
  10. 2025KeyframeFace: Language-Driven Facial Animation via Semantic Keyframescited by 0
  11. 2026Splat-Portrait: Generalizing Talking Heads with Gaussian Splattingcited by 0
  12. 20263DXTalker: Unifying Identity, Lip Sync, Emotion, and Spatial Dynamics in Expressive 3D Talking Avatarscited by 0
17 more
  1. 2026GDPO-Listener: Expressive Interactive Head Generation via Auto-Regressive Flow Matching and Group reward-Decoupled Policy Optimizationcited by 0
  2. 2026PianoFlow: Music-Aware Streaming Piano Motion Generation with Bimanual Coordinationcited by 0
  3. 2026Personalizing Causal Audio-Driven Facial Motion via Dynamic Multi-modal Retrievalcited by 0
  4. 2026OmniRobotHome: A Multi-Camera Home Platform for Real-Time Human-Robot Interactioncited by 0
  5. 2026CapTalk: Text-Guided Stylization and Speech-Driven 3D Head Animationcited by 0
  6. 2026KM-Speaker: Keypoint-Based Style Control for High-Quality Speech-Driven 3D Facial Animation and Dialogue Localizationcited by 0
  7. 2026FacePlex: Toward Natural Full-Duplex Conversational Avatarscited by 0
  8. 2026FFAvatar: Feed-Forward 4D Head Avatar Reconstruction from Sparse Portrait Imagescited by 0
  9. 2026SubtleTalk: Generating Controllable Weakly-correlated Facial Dynamics for 3D Talking Heads via Residual Flow Matchingcited by 0
  10. 2026InterSing: Explicit Interaction Dynamics for 3D Duet Singing Animation and Beyondcited by 0
  11. 2026Decoupled Self-Forcing Distillation for Streaming Talking Head Generationcited by 0
  12. 2026EMODY Flow: Emotion-Aware Audio-Driven Full-Body Motion Generationcited by 0
  13. 2026ComplexSync: High-Fidelity and Real-Time Lip Sync in Complex Scenarioscited by 0
  14. 2026Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animationcited by 0
  15. 2026Beyond End-to-End Black Box Mapping: An Intentional Agent Framework for Cognitive-driven Facial Reaction Generationcited by 0
  16. 2026EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfoldcited by 0
  17. 2026The Shape of Speech: A Geometric Measure of Coarticulation for Speech-Driven 3D Facial Animationcited by 0