# Improving Few-Shot Learning for Talking Face System with TTS Data Augmentation

arXiv: https://arxiv.org/abs/2303.05322 · HTML: https://mlsanity.com/paper/2303.05322/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2023.

## Built on by (most cited first, 6 of 6)

- [AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding](https://mlsanity.com/paper/2405.03121/) (2024)
- [DiffDub: Person-generic Visual Dubbing Using Inpainting Renderer with Diffusion Auto-encoder](https://mlsanity.com/paper/2311.01811/) (2023)
- [Takin-ADA: Emotion Controllable Audio-Driven Animation with Canonical and Landmark Loss Optimization](https://mlsanity.com/paper/2410.14283/) (2024)
- [Generative Data Augmentation Challenge: Zero-Shot Speech Synthesis for Personalized Speech Enhancement](https://mlsanity.com/paper/2501.13372/) (2025)
- [See the Speaker: Crafting High-Resolution Talking Faces from Speech with Prior Guidance and Region Refinement](https://mlsanity.com/paper/2510.26819/) (2025)
- [PS-TTS: Phonetic Synchronization in Text-to-Speech for Achieving Natural Automated Dubbing](https://mlsanity.com/paper/2604.09111/) (2026)

## Abstract (arXiv)

Audio-driven talking face has attracted broad interest from academia and industry recently. However, data acquisition and labeling in audio-driven talking face are labor-intensive and costly. The lack of data resource results in poor synthesis effect. To alleviate this issue, we propose to use TTS (Text-To-Speech) for data augmentation to improve few-shot ability of the talking face system. The misalignment problem brought by the TTS audio is solved with the introduction of soft-DTW, which is first adopted in the talking face task. Moreover, features extracted by HuBERT are explored to utilize underlying information of audio, and found to be superior over other features. The proposed method achieves 17%, 14%, 38% dominance on MSE score, DTW score and user study preference repectively over the baseline model, which shows the effectiveness of improving few-shot learning for talking face system with TTS augmentation.