# Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation

arXiv: https://arxiv.org/abs/1904.04169 · HTML: https://mlsanity.com/paper/1904.04169/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2019.

## Built on by (most cited first, 22 of 22)

- [Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech](https://mlsanity.com/paper/2109.06952/) (2021)
- [Injecting Text in Self-Supervised Speech Pretraining](https://mlsanity.com/paper/2108.12226/) (2021)
- [Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representations](https://mlsanity.com/paper/2303.01664/) (2023)
- [Adversarially learning disentangled speech representations for robust multi-factor voice conversion](https://mlsanity.com/paper/2102.00184/) (2021)
- [Personalized Automatic Speech Recognition Trained on Small Disordered Speech Datasets](https://mlsanity.com/paper/2110.04612/) (2021)
- [Towards Identity Preserving Normal to Dysarthric Voice Conversion](https://mlsanity.com/paper/2110.08213/) (2021)
- [The Effectiveness of Time Stretching for Enhancing Dysarthric Speech for Improved Dysarthric Speech Recognition](https://mlsanity.com/paper/2201.04908/) (2022)
- [UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit Normalization](https://mlsanity.com/paper/2401.14664/) (2024)
- [A Preliminary Study of a Two-Stage Paradigm for Preserving Speaker Identity in Dysarthric Voice Conversion](https://mlsanity.com/paper/2106.01415/) (2021)
- [Global Rhythm Style Transfer Without Text Transcriptions](https://mlsanity.com/paper/2106.08519/) (2021)
- [Replay Attacks Against Audio Deepfake Detection](https://mlsanity.com/paper/2505.14862/) (2025)
- [Transfer the linguistic representations from TTS to accent conversion with non-parallel data](https://mlsanity.com/paper/2401.03538/) (2024)
- [Easy, Interpretable, Effective: openSMILE for voice deepfake detection](https://mlsanity.com/paper/2408.15775/) (2024)
- [Speaker Identity Preservation in Dysarthric Speech Reconstruction by Adversarial Speaker Adaptation](https://mlsanity.com/paper/2202.09082/) (2022)
- [Finding My Voice: Generative Reconstruction of Disordered Speech for Automated Clinical Evaluation](https://mlsanity.com/paper/2509.19231/) (2025)
- [Non-autoregressive real-time Accent Conversion model with voice cloning](https://mlsanity.com/paper/2405.13162/) (2024)
- [Enhancing Zero-Shot Many to Many Voice Conversion with Self-Attention VAE](https://mlsanity.com/paper/2203.16037/) (2022)
- [Correcting Mispronunciations in Speech using Spectrogram Inpainting](https://mlsanity.com/paper/2204.03379/) (2022)
- [DeePen: Penetration Testing for Audio Deepfake Detection](https://mlsanity.com/paper/2502.20427/) (2025)
- [DisC-VC: Disentangled and F0-Controllable Neural Voice Conversion](https://mlsanity.com/paper/2210.11059/) (2022)
- [A New Approach to Voice Authenticity](https://mlsanity.com/paper/2402.06304/) (2024)
- [Synthetic Audio Generation Framework for Air Traffic Control Speech Recognition](https://mlsanity.com/paper/2606.21340/) (2026)

## Abstract (arXiv)

We describe Parrotron, an end-to-end-trained speech-to-speech conversion model that maps an input spectrogram directly to another spectrogram, without utilizing any intermediate discrete representation. The network is composed of an encoder, spectrogram and phoneme decoders, followed by a vocoder to synthesize a time-domain waveform. We demonstrate that this model can be trained to normalize speech from any speaker regardless of accent, prosody, and background noise, into the voice of a single canonical target speaker with a fixed accent and consistent articulation and prosody. We further show that this normalization model can be adapted to normalize highly atypical speech from a deaf speaker, resulting in significant improvements in intelligibility and naturalness, measured via a speech recognizer and listening tests. Finally, demonstrating the utility of this model on other speech tasks, we show that the same model architecture can be trained to perform a speech separation task