# Voice Conversion for Whispered Speech Synthesis

arXiv: https://arxiv.org/abs/1912.05289 · HTML: https://mlsanity.com/paper/1912.05289/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2019.

## Built on by (most cited first, 7 of 7)

- [Towards Identity Preserving Normal to Dysarthric Voice Conversion](https://mlsanity.com/paper/2110.08213/) (2021)
- [Whispered and Lombard Neural Speech Synthesis](https://mlsanity.com/paper/2101.05313/) (2021)
- [Voicy: Zero-Shot Non-Parallel Voice Conversion in Noisy Reverberant Environments](https://mlsanity.com/paper/2106.08873/) (2021)
- [DualVoice: Speech Interaction that Discriminates between Normal and Whispered Voice Input](https://mlsanity.com/paper/2208.10499/) (2022)
- [Quartered Spectral Envelope and 1D-CNN-based Classification of Normally Phonated and Whispered Speech](https://mlsanity.com/paper/2408.13746/) (2024)
- [Collective Learning Mechanism based Optimal Transport Generative Adversarial Network for Non-parallel Voice Conversion](https://mlsanity.com/paper/2504.13791/) (2025)
- [DeepASMR: LLM-Based Zero-Shot ASMR Speech Generation for Anyone of Any Voice](https://mlsanity.com/paper/2601.15596/) (2026)

## Abstract (arXiv)

We present an approach to synthesize whisper by applying a handcrafted signal processing recipe and Voice Conversion (VC) techniques to convert normally phonated speech to whispered speech. We investigate using Gaussian Mixture Models (GMM) and Deep Neural Networks (DNN) to model the mapping between acoustic features of normal speech and those of whispered speech. We evaluate naturalness and speaker similarity of the converted whisper on an internal corpus and on the publicly available wTIMIT corpus. We show that applying VC techniques is significantly better than using rule-based signal processing methods and it achieves results that are indistinguishable from copy-synthesis of natural whisper recordings. We investigate the ability of the DNN model to generalize on unseen speakers, when trained with data from multiple speakers. We show that excluding the target speaker from the training set has little or no impact on the perceived naturalness and speaker similarity of the converted whisper. The proposed DNN method is used in the newly released Whisper Mode of Amazon Alexa.