mlsanityData to 14 Oct 2026

Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens

arXiv 1910.11997 ↗First posted 26 Oct 2019cs.SD cs.LG eess.AScited by 44

Not on the map. Why?

Abstract (arXiv)

Mellotron is a multispeaker voice synthesis model based on Tacotron 2 GST that can make a voice emote and sing without emotive or singing training data. By explicitly conditioning on rhythm and continuous pitch contours from an audio signal or music score, Mellotron is able to generate speech in a variety of styles ranging from read speech to expressive speech, from slow drawls to rap and from monotonous voice to singing voice. Unlike other methods, we train Mellotron using only read speech data without alignments between text and audio. We evaluate our models using the LJSpeech and LibriTTS datasets. We provide F0 Frame Errors and synthesized samples that include style transfer from other speakers, singers and styles not seen during training, procedural manipulation of rhythm and pitch and choir synthesis.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 20; 2022: 12; 2023: 5; 2024: 5; 2025: 1; 2026: 1.

44 papers.

Sort:

By area:

  1. 2021A Survey on Neural Speech Synthesiscited by 184
  2. 2022StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesiscited by 33
  3. 2021V2C: Visual Voice Cloningcited by 24
  4. 2024Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignmentcited by 22
  5. 2022GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speechcited by 21
  6. 2022Controllable Data Generation by Deep Learning: A Reviewcited by 20
  7. 2024EmoSphere++: Emotion-Controllable Zero-Shot Text-to-Speech via Emotion-Adaptive Spherical Vectorcited by 19
  8. 2021STYLER: Style Factor Modeling with Rapidity and Robustness via Speech Decomposition for Expressive and Controllable Neural Text to Speechcited by 18
  9. 2021Adversarially learning disentangled speech representations for robust multi-factor voice conversioncited by 14
  10. 2021Adapting TTS models For New Speakers using Transfer Learningcited by 12
  11. 2021Expressive Neural Voice Cloningcited by 11
  12. 2021Assem-VC: Realistic Voice Conversion by Assembling Modern Speech Synthesis Techniquescited by 11
32 more
  1. 2021Style Equalization: Unsupervised Learning of Controllable Generative Sequence Modelscited by 11
  2. 2023Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation modelcited by 10
  3. 2022Cross-speaker style transfer for text-to-speech using data augmentationcited by 9
  4. 2022SpecRNet: Towards Faster and More Accessible Audio DeepFake Detectioncited by 9
  5. 2022Cross-Speaker Emotion Transfer for Low-Resource Text-to-Speech Using Non-Parallel Voice Conversion with Pitch-Shift Data Augmentationcited by 8
  6. 2022NatiQ: An End-to-end Text-to-Speech System for Arabiccited by 7
  7. 2021Review of end-to-end speech synthesis technology based on deep learningcited by 6
  8. 2021Exploring emotional prototypes in a high dimensional TTS latent spacecited by 6
  9. 2021Global Rhythm Style Transfer Without Text Transcriptionscited by 6
  10. 2021Zero-shot Voice Conversion via Self-supervised Prosody Representation Learningcited by 6
  11. 2021RefineGAN: Universally Generating Waveform Better than Ground Truth with Highly Accurate Pitch and Intensity Responsescited by 5
  12. 2022A Text-to-Speech Pipeline, Evaluation Methodology, and Initial Fine-Tuning Results for Child Speech Synthesiscited by 5
  13. 2024DEX-TTS: Diffusion-based EXpressive Text-to-Speech with Style Modeling on Time Variabilitycited by 5
  14. 2021Facetron: A Multi-speaker Face-to-Speech Model based on Cross-modal Latent Representationscited by 4
  15. 2021Improved Prosodic Clustering for Multispeaker and Speaker-independent Phoneme-level Prosody Controlcited by 3
  16. 2022SingAug: Data Augmentation for Singing Voice Synthesis with Cycle-consistent Training Strategycited by 3
  17. 2022Controllable speech synthesis by learning discrete phoneme-level prosodic representationscited by 3
  18. 2024Disentangling segmental and prosodic factors to non-native speech comprehensibilitycited by 3
  19. 2021GANtron: Emotional Speech Synthesis with Generative Adversarial Networkscited by 2
  20. 2021Rapping-Singing Voice Synthesis based on Phoneme-level Prosody Controlcited by 2
  21. 2022Generative Modeling for Low Dimensional Speech Attributes with Neural Spline Flowscited by 2
  22. 2023An investigation of phrase break prediction in an End-to-End TTS systemcited by 2
  23. 2023Controllable Speaking Styles Using a Large Language Modelcited by 2
  24. 2022Karaoker: Alignment-free singing voice synthesis with speech training datacited by 1
  25. 2023A Discourse-level Multi-scale Prosodic Model for Fine-grained Emotion Analysiscited by 1
  26. 2021Synchronising speech segments with musical beats in Mandarin and English singingcited by 0
  27. 2021A Deep-Bayesian Framework for Adaptive Speech Duration Modificationcited by 0
  28. 2021Pitch Preservation In Singing Voice Synthesiscited by 0
  29. 2023Improved Child Text-to-Speech Synthesis through Fastpitch-based Transfer Learningcited by 0
  30. 2024Re-ENACT: Reinforcement Learning for Emotional Speech Generation using Actor-Critic Strategycited by 0
  31. 2025VisualSpeech: Enhancing Prosody Modeling in TTS Using Videocited by 0
  32. 2026Linguistically Augmented Audio Speech Data (LinguAS)cited by 0