mlsanityData to 14 Oct 2026

MemoryTalker: Personalized Speech-Driven 3D Facial Animation via Audio-Guided Stylization

arXiv 2507.20562 ↗First posted 28 Jul 2025cs.CV cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Speech-driven 3D facial animation aims to synthesize realistic facial motion sequences from given audio, matching the speaker's speaking style. However, previous works often require priors such as class labels of a speaker or additional 3D facial meshes at inference, which makes them fail to reflect the speaking style and limits their practical use. To address these issues, we propose MemoryTalker which enables realistic and accurate 3D facial motion synthesis by reflecting speaking style only with audio input to maximize usability in applications. Our framework consists of two training stages: 1-stage is storing and retrieving general motion (i.e., Memorizing), and 2-stage is to perform the personalized facial motion synthesis (i.e., Animating) with the motion memory stylized by the audio-driven speaking style feature. In this second stage, our model learns about which facial motion types should be emphasized for a particular piece of audio. As a result, our MemoryTalker can generate a reliable personalized facial animation without additional prior information. With quantitative and qualitative evaluations, as well as user study, we show the effectiveness of our model and its performance enhancement for personalized facial animation over state-of-the-art methods.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 28 Jul 2025.

Builds on

37 of 54 references.

References · 37

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  2. 2019Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detectioncited by 564
  3. 2014Memory Networkscited by 363
  4. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 259
  5. 2016Key-Value Memory Networks for Directly Reading Documentscited by 223
  6. 2021FaceFormer: Speech-Driven 3D Facial Animation with Transformerscited by 209
  7. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 190
  8. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 184
  9. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 173
  10. 2023EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animationcited by 142
  11. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 105
  12. 2020Audio-driven Talking Face Video Generation with Learning-based Personalized Head Posecited by 92
25 more
  1. 2021STMTrack: Template-free Visual Tracking with Space-time Memory Networkscited by 91
  2. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Modelscited by 81
  3. 2022Imitator: Personalized Speech-driven 3D Facial Animationcited by 74
  4. 2023SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Facescited by 68
  5. 2018Memory Matching Networks for One-Shot Image Recognitioncited by 65
  6. 2022SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memorycited by 65
  7. 2021Video Prediction Recalling Long-term Motion Context via Memory Alignment Learningcited by 62
  8. 2018Learning Dynamic Memory Networks for Object Trackingcited by 59
  9. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55
  10. 2022Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Readingcited by 39
  11. 2024Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidancecited by 39
  12. 2024UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Modelcited by 30
  13. 2024ScanTalk: 3D Talking Heads from Unregistered Scanscited by 24
  14. 2020Geometry-guided Dense Perspective Network for Speech-Driven Facial Animationcited by 19
  15. 2022Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Videocited by 18
  16. 2023Speech-Driven 3D Face Animation with Composite and Regional Facial Movementscited by 17
  17. 2023Graph-level Anomaly Detection via Hierarchical Memory Networkscited by 16
  18. 2023Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animationcited by 16
  19. 2024Learn2Talk: 3D Talking Face Learns from 2D Talking Facecited by 16
  20. 2023Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applicationscited by 15
  21. 2024KMTalk: Speech-Driven 3D Facial Animation with Key Motion Embeddingcited by 14
  22. 2021Disentanglement of Emotional Style and Speaker Identity for Expressive Voice Conversioncited by 13
  23. 2024Enhancing Speech-Driven 3D Facial Animation with Audio-Visual Guidance from Lip Reading Expertcited by 6
  24. 2023AVFace: Towards Detailed Audio-Visual 4D Face Reconstructioncited by 4
  25. 2023Personalized Speech-driven Expressive 3D Facial Animation Synthesis with Style Controlcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Talking Together: Synthesizing Co-Located 3D Conversations from Audiocited by 0
  2. 2026FreeTalk: Emotional Topology-Free 3D Talking Headscited by 0
  3. 2026PianoFlow: Music-Aware Streaming Piano Motion Generation with Bimanual Coordinationcited by 0
  4. 2026Personalizing Causal Audio-Driven Facial Motion via Dynamic Multi-modal Retrievalcited by 0
  5. 2026ArtBoost: Synthetic Articulatory Data Augmentation for Acoustic-to-Articulatory Inversioncited by 0
  6. 2026SubtleTalk: Generating Controllable Weakly-correlated Facial Dynamics for 3D Talking Heads via Residual Flow Matchingcited by 0
  7. 2026ECHO: Dyadic 3D Facial Motion Generation with Asymmetric Deterministic Articulation and Stochastic Reactioncited by 0
  8. 2026KoUniTalk: A Lightweight Articulation-Centered Korean-English 3D Talking Face Benchmarkcited by 0
  9. 2026Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animationcited by 0