mlsanityData to 14 Oct 2026

R-MelNet: Reduced Mel-Spectral Modeling for Neural TTS

arXiv 2206.15276 ↗First posted 30 Jun 2022cs.SD cs.LG eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

This paper introduces R-MelNet, a two-part autoregressive architecture with a frontend based on the first tier of MelNet and a backend WaveRNN-style audio decoder for neural text-to-speech synthesis. Taking as input a mixed sequence of characters and phonemes, with an optional audio priming sequence, this model produces low-resolution mel-spectral features which are interpolated and used by a WaveRNN decoder to produce an audio waveform. Coupled with half precision training, R-MelNet uses under 11 gigabytes of GPU memory on a single commodity GPU (NVIDIA 2080Ti). We detail a number of critical implementation details for stable half precision training, including an approximate, numerically stable mixture of logistics attention. Using a stochastic, multi-sample per step inference scheme, the resulting model generates highly varied audio, while enabling text and audio based controls to modify output waveforms. Qualitative and quantitative evaluations of an R-MelNet system trained on a single speaker TTS dataset demonstrate the effectiveness of our approach.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 30 Jun 2022.

Builds on

31 of 42 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  5. 2016Density estimation using Real NVPcited by 2,042
  6. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  8. 2016Pixel Recurrent Neural Networkscited by 816
  9. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  10. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  11. 2013Generating Sequences With Recurrent Neural Networkscited by 554
  12. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
19 more
  1. 2018Efficient Neural Audio Synthesiscited by 219
  2. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  3. 2018Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscalingcited by 74
  4. 2021WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesiscited by 65
  5. 2022The VoiceMOS Challenge 2022cited by 57
  6. 2021PortaSpeech: Portable and High-Quality Generative Text-to-Speechcited by 47
  7. 2019Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesiscited by 44
  8. 2019MelNet: A Generative Model for Audio in the Frequency Domaincited by 42
  9. 2021Chunked Autoregressive GAN for Conditional Waveform Synthesiscited by 38
  10. 2019Augment your batch: better training with larger batchescited by 37
  11. 2021Bilateral Denoising Diffusion Modelscited by 36
  12. 2015Generative Image Modeling Using Spatial LSTMscited by 35
  13. 2015ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networkscited by 24
  14. 2015Grid Long Short-Term Memorycited by 20
  15. 2020I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitchcited by 17
  16. 2021Speaker Generationcited by 14
  17. 2021Style Equalization: Unsupervised Learning of Controllable Generative Sequence Modelscited by 11
  18. 2021Neural HMMs are all you need (for high-quality attention-free TTS)cited by 8
  19. 2018Representation Mixing for TTS Synthesiscited by 7

Built on by

Not cited yet.