# Multi-SpectroGAN: High-Diversity and High-Fidelity Spectrogram Generation with Adversarial Style Combination for Speech Synthesis

arXiv: https://arxiv.org/abs/2012.07267 · HTML: https://mlsanity.com/paper/2012.07267/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2020.

## Built on by (most cited first, 28 of 28)

- [Diffsound: Discrete Diffusion Model for Text-to-sound Generation](https://mlsanity.com/paper/2207.09983/) (2022)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion](https://mlsanity.com/paper/2305.15816/) (2023)
- [HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesis](https://mlsanity.com/paper/2311.12454/) (2023)
- [MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling](https://mlsanity.com/paper/2112.09312/) (2021)
- [Fre-GAN: Adversarial Frequency-consistent Audio Synthesis](https://mlsanity.com/paper/2106.02297/) (2021)
- [Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2412.06602/) (2024)
- [ProsoSpeech: Enhancing Prosody With Quantized Vector Pre-training in Text-to-Speech](https://mlsanity.com/paper/2202.07816/) (2022)
- [Faces that Speak: Jointly Synthesising Talking Face and Speech from Text](https://mlsanity.com/paper/2405.10272/) (2024)
- [DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech Synthesis with Prosody Conditional Adversarial Training](https://mlsanity.com/paper/2307.16549/) (2023)
- [Let There Be Sound: Reconstructing High Quality Speech from Silent Videos](https://mlsanity.com/paper/2308.15256/) (2023)
- [VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglement](https://mlsanity.com/paper/2211.10666/) (2022)
- [HierVST: Hierarchical Adaptive Zero-shot Voice Style Transfer](https://mlsanity.com/paper/2307.16171/) (2023)
- [DurFlex-EVC: Duration-Flexible Emotional Voice Conversion Leveraging Discrete Representations without Text Alignment](https://mlsanity.com/paper/2401.08095/) (2024)
- [PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-based Prosody Modeling](https://mlsanity.com/paper/2306.07489/) (2023)
- [Accelerating High-Fidelity Waveform Generation via Adversarial Flow Matching Optimization](https://mlsanity.com/paper/2408.08019/) (2024)
- [Voice Cloning: Comprehensive Survey](https://mlsanity.com/paper/2505.00579/) (2025)
- [VoicePrompter: Robust Zero-Shot Voice Conversion with Voice Prompt and Conditional Flow Matching](https://mlsanity.com/paper/2501.17612/) (2025)
- [Controllable Accented Text-to-Speech Synthesis](https://mlsanity.com/paper/2209.10804/) (2022)
- [Semi-supervised learning for continuous emotional intensity controllable speech synthesis with disentangled representations](https://mlsanity.com/paper/2211.06160/) (2022)
- [Multi-modal Adversarial Training for Zero-Shot Voice Cloning](https://mlsanity.com/paper/2408.15916/) (2024)
- [Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesis](https://mlsanity.com/paper/2411.17690/) (2024)
- [GC-TTS: Few-shot Speaker Adaptation with Geometric Constraints](https://mlsanity.com/paper/2108.06890/) (2021)
- [TcGAN: Semantic-Aware and Structure-Preserved GANs with Individual Vision Transformer for Fast Arbitrary One-Shot Image Generation](https://mlsanity.com/paper/2302.08047/) (2023)
- [Chirp Localization via Fine-Tuned Transformer Model: A Proof-of-Concept Study](https://mlsanity.com/paper/2503.22713/) (2025)
- [A Factorization Approach for Motor Imagery Classification](https://mlsanity.com/paper/2112.08175/) (2021)
- [Factorization Approach for Sparse Spatio-Temporal Brain-Computer Interface](https://mlsanity.com/paper/2206.08494/) (2022)
- [Pathway to Future Symbiotic Creativity](https://mlsanity.com/paper/2209.02388/) (2022)

## Abstract (arXiv)

While generative adversarial networks (GANs) based neural text-to-speech (TTS) systems have shown significant improvement in neural speech synthesis, there is no TTS system to learn to synthesize speech from text sequences with only adversarial feedback. Because adversarial feedback alone is not sufficient to train the generator, current models still require the reconstruction loss compared with the ground-truth and the generated mel-spectrogram directly. In this paper, we present Multi-SpectroGAN (MSG), which can train the multi-speaker model with only the adversarial feedback by conditioning a self-supervised hidden representation of the generator to a conditional discriminator. This leads to better guidance for generator training. Moreover, we also propose adversarial style combination (ASC) for better generalization in the unseen speaking style and transcript, which can learn latent representations of the combined style embedding from multiple mel-spectrograms. Trained with ASC and feature matching, the MSG synthesizes a high-diversity mel-spectrogram by controlling and mixing the individual speaking styles (e.g., duration, pitch, and energy). The result shows that the MSG synthesizes a high-fidelity mel-spectrogram, which has almost the same naturalness MOS score as the ground-truth mel-spectrogram.