# Federated Learning with Dynamic Transformer for Text to Speech

arXiv: https://arxiv.org/abs/2107.08795 · HTML: https://mlsanity.com/paper/2107.08795/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 29 references that are in the map.
- First posted on arXiv 9 Jul 2021.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Federated Machine Learning: Concept and Applications](https://mlsanity.com/paper/1902.04885/) (2019)
- [MultiSpeech: Multi-Speaker Text to Speech with Transformer](https://mlsanity.com/paper/2006.04664/) (2020)
- [Training Keyword Spotting Models on Non-IID Data with Federated Learning](https://mlsanity.com/paper/2005.10406/) (2020)
- [AlignTTS: Efficient Feed-Forward Text-to-Speech System without Explicit Alignment](https://mlsanity.com/paper/2003.01950/) (2020)
- [Efficient Client Contribution Evaluation for Horizontal Federated Learning](https://mlsanity.com/paper/2102.13314/) (2021)
- [A Quantitative Metric for Privacy Leakage in Federated Learning](https://mlsanity.com/paper/2102.13472/) (2021)
- [Learning Speaker Embedding from Text-to-Speech](https://mlsanity.com/paper/2010.11221/) (2020)
- [GraphTTS: graph-to-sequence modelling in neural text-to-speech](https://mlsanity.com/paper/2003.01924/) (2020)
- [GraphPB: Graphical Representations of Prosody Boundary in Speech Synthesis](https://mlsanity.com/paper/2012.02626/) (2020)

## Built on by (most cited first, 11 of 11)

- [FedET: A Communication-Efficient Federated Class-Incremental Learning Framework Based on Enhanced Transformer](https://mlsanity.com/paper/2306.15347/) (2023)
- [Visual Transformer Meets CutMix for Improved Accuracy, Communication Efficiency, and Data Privacy in Split Learning](https://mlsanity.com/paper/2207.00234/) (2022)
- [r-G2P: Evaluating and Enhancing Robustness of Grapheme to Phoneme Conversion by Controlled noise introducing and Contextual information incorporation](https://mlsanity.com/paper/2202.11194/) (2022)
- [Adaptive Few-Shot Learning Algorithm for Rare Sound Event Detection](https://mlsanity.com/paper/2205.11738/) (2022)
- [DT-SV: A Transformer-based Time-domain Approach for Speaker Verification](https://mlsanity.com/paper/2205.13249/) (2022)
- [Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition](https://mlsanity.com/paper/2209.15176/) (2022)
- [Privacy-Preserving Split Learning with Vision Transformers using Patch-Wise Random and Noisy CutMix](https://mlsanity.com/paper/2408.01040/) (2024)
- [QSpeech: Low-Qubit Quantum Speech Application Toolkit](https://mlsanity.com/paper/2205.13221/) (2022)
- [Mitigating the Exposure Bias in Sentence-Level Grapheme-to-Phoneme (G2P) Transduction](https://mlsanity.com/paper/2308.08442/) (2023)
- [Learning Invariant Representation and Risk Minimized for Unsupervised Accent Domain Adaptation](https://mlsanity.com/paper/2210.08182/) (2022)
- [Fed-PISA: Federated Voice Cloning via Personalized Identity-Style Adaptation](https://mlsanity.com/paper/2509.16010/) (2025)

## Abstract (arXiv)

Text to speech (TTS) is a crucial task for user interaction, but TTS model training relies on a sizable set of high-quality original datasets. Due to privacy and security issues, the original datasets are usually unavailable directly. Recently, federated learning proposes a popular distributed machine learning paradigm with an enhanced privacy protection mechanism. It offers a practical and secure framework for data owners to collaborate with others, thus obtaining a better global model trained on the larger dataset. However, due to the high complexity of transformer models, the convergence process becomes slow and unstable in the federated learning setting. Besides, the transformer model trained in federated learning is costly communication and limited computational speed on clients, impeding its popularity. To deal with these challenges, we propose the federated dynamic transformer. On the one hand, the performance is greatly improved comparing with the federated transformer, approaching centralize-trained Transformer-TTS when increasing clients number. On the other hand, it achieves faster and more stable convergence in the training phase and significantly reduces communication time. Experiments on the LJSpeech dataset also strongly prove our method's advantage.