# FedSpeech: Federated Text-to-Speech with Continual Learning

arXiv: https://arxiv.org/abs/2110.07216 · HTML: https://mlsanity.com/paper/2110.07216/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2021.

## Built on by (most cited first, 16 of 16)

- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation](https://mlsanity.com/paper/2303.06937/) (2023)
- [Federated Learning on Heterogeneous and Long-Tailed Data via Classifier Re-Training with Federated Features](https://mlsanity.com/paper/2204.13399/) (2022)
- [A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Tasks](https://mlsanity.com/paper/2311.07784/) (2023)
- [Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning](https://mlsanity.com/paper/2409.01128/) (2024)
- [GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech](https://mlsanity.com/paper/2205.07211/) (2022)
- [Personalized Federated Domain-Incremental Learning based on Adaptive Knowledge Matching](https://mlsanity.com/paper/2407.05005/) (2024)
- [Text-Enhanced Data-free Approach for Federated Class-Incremental Learning](https://mlsanity.com/paper/2403.14101/) (2024)
- [FedSampling: A Better Sampling Strategy for Federated Learning](https://mlsanity.com/paper/2306.14245/) (2023)
- [SacFL: Self-Adaptive Federated Continual Learning for Resource-Constrained End Devices](https://mlsanity.com/paper/2505.00365/) (2025)
- [Privacy-preserving Online AutoML for Domain-Specific Face Detection](https://mlsanity.com/paper/2203.08399/) (2022)
- [Don't Memorize; Mimic The Past: Federated Class Incremental Learning Without Episodic Memory](https://mlsanity.com/paper/2307.00497/) (2023)
- [Coordinated Replay Sample Selection for Continual Federated Learning](https://mlsanity.com/paper/2310.15054/) (2023)
- [Dynamic Allocation Hypernetwork with Adaptive Model Recalibration for FCL](https://mlsanity.com/paper/2503.18064/) (2025)
- [Enhancing the Performance of Global Model by Improving the Adaptability of Local Models in Federated Learning](https://mlsanity.com/paper/2505.10125/) (2025)
- [Fed-PISA: Federated Voice Cloning via Personalized Identity-Style Adaptation](https://mlsanity.com/paper/2509.16010/) (2025)

## Abstract (arXiv)

Federated learning enables collaborative training of machine learning models under strict privacy restrictions and federated text-to-speech aims to synthesize natural speech of multiple users with a few audio training samples stored in their devices locally. However, federated text-to-speech faces several challenges: very few training samples from each speaker are available, training samples are all stored in local device of each user, and global model is vulnerable to various attacks. In this paper, we propose a novel federated learning architecture based on continual learning approaches to overcome the difficulties above. Specifically, 1) we use gradual pruning masks to isolate parameters for preserving speakers' tones; 2) we apply selective masks for effectively reusing knowledge from tasks; 3) a private speaker embedding is introduced to keep users' privacy. Experiments on a reduced VCTK dataset demonstrate the effectiveness of FedSpeech: it nearly matches multi-task training in terms of multi-speaker speech quality; moreover, it sufficiently retains the speakers' tones and even outperforms the multi-task training in the speaker similarity experiment.