# MemoryTalker: Personalized Speech-Driven 3D Facial Animation via Audio-Guided Stylization

arXiv: https://arxiv.org/abs/2507.20562 · HTML: https://mlsanity.com/paper/2507.20562/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 54 references that are in the map.
- First posted on arXiv 28 Jul 2025.

## Builds on (most cited first, 30 of 37)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection](https://mlsanity.com/paper/1904.02639/) (2019)
- [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [Key-Value Memory Networks for Directly Reading Documents](https://mlsanity.com/paper/1606.03126/) (2016)
- [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior](https://mlsanity.com/paper/2301.02379/) (2023)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation](https://mlsanity.com/paper/2303.11089/) (2023)
- [EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling](https://mlsanity.com/paper/2401.00374/) (2023)
- [Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose](https://mlsanity.com/paper/2002.10137/) (2020)
- [STMTrack: Template-free Visual Tracking with Space-time Memory Networks](https://mlsanity.com/paper/2104.00324/) (2021)
- [DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models](https://mlsanity.com/paper/2310.00434/) (2023)
- [Imitator: Personalized Speech-driven 3D Facial Animation](https://mlsanity.com/paper/2301.00023/) (2022)
- [SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Faces](https://mlsanity.com/paper/2306.10799/) (2023)
- [Memory Matching Networks for One-Shot Image Recognition](https://mlsanity.com/paper/1804.08281/) (2018)
- [SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory](https://mlsanity.com/paper/2211.00924/) (2022)
- [Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning](https://mlsanity.com/paper/2104.00924/) (2021)
- [Learning Dynamic Memory Networks for Object Tracking](https://mlsanity.com/paper/1803.07268/) (2018)
- [FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models](https://mlsanity.com/paper/2312.08459/) (2023)
- [Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading](https://mlsanity.com/paper/2204.01725/) (2022)
- [Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance](https://mlsanity.com/paper/2401.15687/) (2024)
- [UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified Model](https://mlsanity.com/paper/2408.00762/) (2024)
- [ScanTalk: 3D Talking Heads from Unregistered Scans](https://mlsanity.com/paper/2403.10942/) (2024)
- [Geometry-guided Dense Perspective Network for Speech-Driven Facial Animation](https://mlsanity.com/paper/2008.10004/) (2020)
- [Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video](https://mlsanity.com/paper/2204.01265/) (2022)
- [Speech-Driven 3D Face Animation with Composite and Regional Facial Movements](https://mlsanity.com/paper/2308.05428/) (2023)
- [Graph-level Anomaly Detection via Hierarchical Memory Networks](https://mlsanity.com/paper/2307.00755/) (2023)
- [Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation](https://mlsanity.com/paper/2312.10877/) (2023)

## Built on by (most cited first, 9 of 9)

- [Talking Together: Synthesizing Co-Located 3D Conversations from Audio](https://mlsanity.com/paper/2603.08674/) (2026)
- [FreeTalk: Emotional Topology-Free 3D Talking Heads](https://mlsanity.com/paper/2603.15512/) (2026)
- [PianoFlow: Music-Aware Streaming Piano Motion Generation with Bimanual Coordination](https://mlsanity.com/paper/2604.12856/) (2026)
- [Personalizing Causal Audio-Driven Facial Motion via Dynamic Multi-modal Retrieval](https://mlsanity.com/paper/2604.23692/) (2026)
- [ArtBoost: Synthetic Articulatory Data Augmentation for Acoustic-to-Articulatory Inversion](https://mlsanity.com/paper/2606.16327/) (2026)
- [SubtleTalk: Generating Controllable Weakly-correlated Facial Dynamics for 3D Talking Heads via Residual Flow Matching](https://mlsanity.com/paper/2608.06408/) (2026)
- [ECHO: Dyadic 3D Facial Motion Generation with Asymmetric Deterministic Articulation and Stochastic Reaction](https://mlsanity.com/paper/2609.05506/) (2026)
- [KoUniTalk: A Lightweight Articulation-Centered Korean-English 3D Talking Face Benchmark](https://mlsanity.com/paper/2609.19840/) (2026)
- [Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation](https://mlsanity.com/paper/2609.30517/) (2026)

## Abstract (arXiv)

Speech-driven 3D facial animation aims to synthesize realistic facial motion sequences from given audio, matching the speaker's speaking style. However, previous works often require priors such as class labels of a speaker or additional 3D facial meshes at inference, which makes them fail to reflect the speaking style and limits their practical use. To address these issues, we propose MemoryTalker which enables realistic and accurate 3D facial motion synthesis by reflecting speaking style only with audio input to maximize usability in applications. Our framework consists of two training stages: 1-stage is storing and retrieving general motion (i.e., Memorizing), and 2-stage is to perform the personalized facial motion synthesis (i.e., Animating) with the motion memory stylized by the audio-driven speaking style feature. In this second stage, our model learns about which facial motion types should be emphasized for a particular piece of audio. As a result, our MemoryTalker can generate a reliable personalized facial animation without additional prior information. With quantitative and qualitative evaluations, as well as user study, we show the effectiveness of our model and its performance enhancement for personalized facial animation over state-of-the-art methods.