mlsanityData to 19 Oct 2026

Parallel GPT: Harmonizing the Independence and Interdependence of Acoustic and Semantic Information for Zero-Shot Text-to-Speech

arXiv 2508.04141 ↗First posted 6 Aug 2025eess.AS cs.SDcited by 0

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

Advances in speech representation and large language models have enhanced zero-shot text-to-speech (TTS) performance. However, existing zero-shot TTS models face challenges in capturing the complex correlations between acoustic and semantic features, resulting in a lack of expressiveness and similarity. The primary reason lies in the complex relationship between semantic and acoustic features, which manifests independent and interdependent aspects.This paper introduces a TTS framework that combines both autoregressive (AR) and non-autoregressive (NAR) modules to harmonize the independence and interdependence of acoustic and semantic information. The AR model leverages the proposed Parallel Tokenizer to synthesize the top semantic and acoustic tokens simultaneously. In contrast, considering the interdependence, the Coupled NAR model predicts detailed tokens based on the general AR model's output. Parallel GPT, built on this architecture, is designed to improve zero-shot text-to-speech synthesis through its parallel structure. Experiments on English and Chinese datasets demonstrate that the proposed model significantly outperforms the quality and efficiency of the synthesis of existing zero-shot TTS models. Speech demos are available at https://t1235-ch.github.io/pgpt/.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 43 of its 57 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

43 of 57 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching9 Oct
  2. 2024SoCodec: A Semantic-Ordered Multi-Stream Speech Codec for Efficient Language Model Based Text-to-Speech Synthesis2 Sep
  3. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer1 Sep
  4. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens7 Jul
  5. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS26 Jun
  6. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Models4 Jun
  7. 2024SimpleSpeech: Towards Simple and Efficient Text-to-Speech with Scalar Latent Transformer Diffusion Models4 Jun
  8. 2023Transduce and Speak: Neural Transducer for Text-to-Speech with Semantic Token Prediction6 Nov
  9. 2023Vec-Tok Speech: speech vectorization and tokenization for neural speech generation11 Oct
  10. 2023FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec14 Sep
  11. 2023Better speech synthesis through scaling12 May
  12. 2023HiFi-Codec: Group-residual Vector quantization for High Fidelity Audio Codec4 May
7 more
  1. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers5 Jan
  2. 2022StyleTTS-VC: One-Shot Voice Conversion by Knowledge Transfer from Style-Based TTS Models29 Dec
  3. 2022StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis30 May
  4. 2022JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech31 Mar
  5. 2022Transfer Learning Framework for Low-Resource Text-to-Speech using a Large-Scale Unlabeled Speech Corpus29 Mar
  6. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone4 Dec
  7. 2021SoundStream: An End-to-End Neural Audio Codec7 Jul

Other references · 24

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,267
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,599
  3. 2022Flow Matching for Generative Modelingcited by 5,905
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 4,385
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,762
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,880
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,462
  8. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,338
  9. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 1,023
  10. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 980
  11. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 947
  12. 2022High Fidelity Neural Audio Compressioncited by 917
12 more
  1. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 864
  2. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 711
  3. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 619
  4. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 552
  5. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 422
  6. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 412
  7. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 268
  8. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 244
  9. 2023CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Maskingcited by 162
  10. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 162
  11. 2024SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metricscited by 68
  12. 2024Generative Expressive Conversational Speech Synthesiscited by 21

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025BridgeCode: A Dual Speech Representation Paradigm for Autoregressive Zero-Shot Text-to-Speech Synthesis
  2. 2025Towards Lightweight and Stable Zero-shot TTS with Self-distilled Representation Disentanglement
  3. 2024MultiVerse: Efficient and Expressive Zero-Shot Multi-Task Text-to-Speech
  4. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias
  5. 2025RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speech
  6. 2026SemaVoice: Semantic-Aware Continuous Autoregressive Speech Synthesis
  7. 2024SpeechGPT-Gen: Scaling Chain-of-Information Speech Generation
  8. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model