# KM-Speaker: Keypoint-Based Style Control for High-Quality Speech-Driven 3D Facial Animation and Dialogue Localization

arXiv: https://arxiv.org/abs/2606.28568 · HTML: https://mlsanity.com/paper/2606.28568/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 26 Jun 2026.

## Builds on (most cited first, 28 of 28)

- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Learning an Animatable Detailed 3D Face Model from In-The-Wild Images](https://mlsanity.com/paper/2012.04012/) (2020)
- [Flow Matching Guide and Code](https://mlsanity.com/paper/2412.06264/) (2024)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [CelebV-HQ: A Large-Scale Video Facial Attributes Dataset](https://mlsanity.com/paper/2207.12393/) (2022)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior](https://mlsanity.com/paper/2301.02379/) (2023)
- [Emotional Speech-Driven Animation with Content-Emotion Disentanglement](https://mlsanity.com/paper/2306.08990/) (2023)
- [Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation](https://mlsanity.com/paper/2106.03153/) (2021)
- [3D face reconstruction with dense landmarks](https://mlsanity.com/paper/2204.02776/) (2022)
- [Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance](https://mlsanity.com/paper/2401.15687/) (2024)
- [ARTalk: Speech-Driven 3D Head Animation via Autoregressive Model](https://mlsanity.com/paper/2502.20323/) (2025)
- [MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Sampling](https://mlsanity.com/paper/2410.10122/) (2024)
- [Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation](https://mlsanity.com/paper/2312.10877/) (2023)
- [MMHead: Towards Fine-grained Multi-modal 3D Facial Animation](https://mlsanity.com/paper/2410.07757/) (2024)
- [KMTalk: Speech-Driven 3D Facial Animation with Key Motion Embedding](https://mlsanity.com/paper/2409.01113/) (2024)
- [ExpCLIP: Bridging Text and Facial Expressions via Semantic Alignment](https://mlsanity.com/paper/2308.14448/) (2023)
- [ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAE](https://mlsanity.com/paper/2409.07966/) (2024)
- [Cafe-Talk: Generating 3D Talking Face Animation with Multimodal Coarse- and Fine-grained Control](https://mlsanity.com/paper/2503.14517/) (2025)
- [Learning Landmarks Motion from Speech for Speaker-Agnostic 3D Talking Heads Generation](https://mlsanity.com/paper/2306.01415/) (2023)
- [EmoFace: Audio-driven Emotional 3D Face Animation](https://mlsanity.com/paper/2407.12501/) (2024)
- [Model See Model Do: Speech-Driven Facial Animation with Style Control](https://mlsanity.com/paper/2505.01319/) (2025)
- [MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding](https://mlsanity.com/paper/2507.06071/) (2025)
- [Towards High-fidelity 3D Talking Avatar with Personalized Dynamic Texture](https://mlsanity.com/paper/2503.00495/) (2025)
- [SEREP: Semantic Facial Expression Representation for Robust In-the-Wild Capture and Retargeting](https://mlsanity.com/paper/2412.14371/) (2024)
- [Identity-Preserving Video Dubbing Using Motion Warping](https://mlsanity.com/paper/2501.04586/) (2025)

## Abstract (arXiv)

Speech-driven 3D facial animation methods face significant challenges in simultaneously achieving high-fidelity motion and precise artistic control at production quality. Existing controllable models typically learn global style control by relying on large-scale, low-quality in-the-wild datasets that compromise overall animation realism. Furthermore, these frameworks often lack the fine-grained temporal precision required for demanding tasks such as dialogue localization (e.g., dubbing), where matching specific facial expressions is as critical as lip synchronization. We present KM-Speaker (Keypoint-Matching Speaker), a novel keypoint-conditioned flow-based generative framework that provides both global style guidance and frame-level temporal control from reference performances. We propose a disentanglement strategy that separates audio-driven lip motion from keypoint-driven upper-face dynamics, together with a global style context preservation mechanism to ensure coherent full-face expressiveness. KM-Speaker advances example-based 3D facial animation by achieving high-fidelity motion and flexible controllability in a data-constrained setting, consistently outperforming state-of-the-art methods in lip-sync accuracy, style adherence, and expressive temporal control.