# Transformer-S2A: Robust and Efficient Speech-to-Animation

arXiv: https://arxiv.org/abs/2111.09771 · HTML: https://mlsanity.com/paper/2111.09771/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts.
- First posted on arXiv 18 Nov 2021.

## Placed in

- Mixture-of-experts / [Multimodal MoE models](https://mlsanity.com/moe/multimodal-moe-models/)

## Built on by (most cited first, 10 of 10)

- [EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation](https://mlsanity.com/paper/2303.11089/) (2023)
- [GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation](https://mlsanity.com/paper/2305.00787/) (2023)
- [Memories are One-to-Many Mapping Alleviators in Talking Face Generation](https://mlsanity.com/paper/2212.05005/) (2022)
- [ExpCLIP: Bridging Text and Facial Expressions via Semantic Alignment](https://mlsanity.com/paper/2308.14448/) (2023)
- [StableFace: Analyzing and Improving Motion Stability for Talking Face Generation](https://mlsanity.com/paper/2208.13717/) (2022)
- [NPVForensics: Jointing Non-critical Phonemes and Visemes for Deepfake Detection](https://mlsanity.com/paper/2306.06885/) (2023)
- [PoseTalk: Text-and-Audio-based Pose Control and Motion Refinement for One-Shot Talking Head Generation](https://mlsanity.com/paper/2409.02657/) (2024)
- [AdaMesh: Personalized Facial Expressions and Head Poses for Adaptive Speech-Driven 3D Facial Animation](https://mlsanity.com/paper/2310.07236/) (2023)
- [FlexLip: A Controllable Text-to-Lip System](https://mlsanity.com/paper/2206.03206/) (2022)
- [Towards Unified Co-Speech Gesture Generation via Hierarchical Implicit Periodicity Learning](https://mlsanity.com/paper/2512.13131/) (2025)

## Abstract (arXiv)

We propose a novel robust and efficient Speech-to-Animation (S2A) approach for synchronized facial animation generation in human-computer interaction. Compared with conventional approaches, the proposed approach utilizes phonetic posteriorgrams (PPGs) of spoken phonemes as input to ensure the cross-language and cross-speaker ability, and introduces corresponding prosody features (i.e. pitch and energy) to further enhance the expression of generated animation. Mixture-of-experts (MOE)-based Transformer is employed to better model contextual information while provide significant optimization on computation efficiency. Experiments demonstrate the effectiveness of the proposed approach on both objective and subjective evaluation with 17x inference speedup compared with the state-of-the-art approach.