# Generative Semantic Communication for Text-to-Speech Synthesis

arXiv: https://arxiv.org/abs/2410.03459 · HTML: https://mlsanity.com/paper/2410.03459/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 19 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 17 of 17)

- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://mlsanity.com/paper/2105.06337/) (2021)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers](https://mlsanity.com/paper/2304.09116/) (2023)
- [Robust Semantic Communications with Masked VQ-VAE Enabled Codebook](https://mlsanity.com/paper/2206.04011/) (2022)
- [What is Semantic Communication? A View on Conveying Meaning in the Era of Machine Intelligence](https://mlsanity.com/paper/2110.00196/) (2021)
- [Towards Semantic Communications: Deep Learning-Based Image Semantic Coding](https://mlsanity.com/paper/2208.04094/) (2022)
- [Deep Learning Enabled Semantic Communications with Speech Recognition and Synthesis](https://mlsanity.com/paper/2205.04603/) (2022)
- [Performance Optimization for Semantic Communications: An Attention-based Reinforcement Learning Approach](https://mlsanity.com/paper/2208.08239/) (2022)
- [Transformer-aided Wireless Image Transmission with Channel Feedback](https://mlsanity.com/paper/2306.09101/) (2023)
- [A Robust Deep Learning Enabled Semantic Communication System for Text](https://mlsanity.com/paper/2206.02596/) (2022)
- [Knowledge Base Enabled Semantic Communication: A Generative Perspective](https://mlsanity.com/paper/2311.12443/) (2023)
- [Semantic Change Driven Generative Semantic Communication Framework](https://mlsanity.com/paper/2309.12775/) (2023)
- [Codebook-enabled Generative End-to-end Semantic Communication Powered by Transformer](https://mlsanity.com/paper/2402.16868/) (2024)

## Built on by (most cited first, 2 of 2)

- [Generative Semantic Communication for Joint Image Transmission and Segmentation](https://mlsanity.com/paper/2411.18005/) (2024)
- [Generative Semantic Communication: Architectures, Technologies, and Applications](https://mlsanity.com/paper/2412.08642/) (2024)

## Abstract (arXiv)

Semantic communication is a promising technology to improve communication efficiency by transmitting only the semantic information of the source data. However, traditional semantic communication methods primarily focus on data reconstruction tasks, which may not be efficient for emerging generative tasks such as text-to-speech (TTS) synthesis. To address this limitation, this paper develops a novel generative semantic communication framework for TTS synthesis, leveraging generative artificial intelligence technologies. Firstly, we utilize a pre-trained large speech model called WavLM and the residual vector quantization method to construct two semantic knowledge bases (KBs) at the transmitter and receiver, respectively. The KB at the transmitter enables effective semantic extraction, while the KB at the receiver facilitates lifelike speech synthesis. Then, we employ a transformer encoder and a diffusion model to achieve efficient semantic coding without introducing significant communication overhead. Finally, numerical results demonstrate that our framework achieves much higher fidelity for the generated speech than four baselines, in both cases with additive white Gaussian noise channel and Rayleigh fading channel.