mlsanityData to 7 Oct 2026

SpeechGPT-Gen: Scaling Chain-of-Information Speech Generation

arXiv 2401.13527 ↗First posted 24 Jan 2024cs.CL cs.SD eess.AScited by 36

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Benefiting from effective speech modeling, current Speech Large Language Models (SLLMs) have demonstrated exceptional capabilities in in-context speech generation and efficient generalization to unseen speakers. However, the prevailing information modeling process is encumbered by certain redundancies, leading to inefficiencies in speech generation. We propose Chain-of-Information Generation (CoIG), a method for decoupling semantic and perceptual information in large-scale speech generation. Building on this, we develop SpeechGPT-Gen, an 8-billion-parameter SLLM efficient in semantic and perceptual information modeling. It comprises an autoregressive model based on LLM for semantic information modeling and a non-autoregressive model employing flow matching for perceptual information modeling. Additionally, we introduce the novel approach of infusing semantic information into the prior distribution to enhance the efficiency of flow matching. Extensive experimental results demonstrate that SpeechGPT-Gen markedly excels in zero-shot text-to-speech, zero-shot voice conversion, and speech-to-speech dialogue, underscoring CoIG's remarkable proficiency in capturing and modeling speech's semantic and perceptual dimensions. Code and models are available at https://github.com/0nutation/SpeechGPT.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 15 of its 21 references that are in the map. First posted on arXiv 24 Jan 2024.

Builds on

15 of 21 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Generative Pre-training for Speech with Flow Matching25 Oct

Other references · 14

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  9. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
  10. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  11. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  12. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
2 more
  1. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  2. 2020Unsupervised Speech Decomposition via Triple Information Bottleneckcited by 87

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 15; 2026: 9.

36 papers.

Sort:

By area:

  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  2. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  3. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  4. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 88
  5. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  6. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 75
  7. 2024Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMscited by 48
  8. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 42
  9. 2025XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecscited by 21
  10. 2024DFADD: The Diffusion and Flow-Matching Based Audio Deepfake Datasetcited by 20
  11. 2024IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilitiescited by 16
  12. 2024Body of Her: A Preliminary Study on End-to-End Humanoid Agentcited by 15
24 more
  1. 2025FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systemscited by 15
  2. 2025Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representationcited by 15
  3. 2025OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interactioncited by 14
  4. 2024PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systemscited by 13
  5. 2026MOSS-TTS Technical Reportcited by 10
  6. 2024Fine-tuning Multimodal Large Language Models for Product Bundlingcited by 9
  7. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  8. 2025FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matchingcited by 8
  9. 2025OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Modelcited by 8
  10. 2024Style-Talker: Finetuning Audio Language Model and Style-Based Text-to-Speech Model for Fast Spoken Dialogue Generationcited by 7
  11. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  12. 2025Chain-of-Thought Training for Open E2E Spoken Dialogue Systemscited by 5
  13. 2026Interpretable All-Type Audio Deepfake Detection with Audio LLMs via Frequency-Time Reinforcement Learningcited by 5
  14. 2026Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedbackcited by 5
  15. 2025Chain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systemscited by 4
  16. 2026VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Modelscited by 3
  17. 2024Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLMcited by 2
  18. 2025SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Modelscited by 2
  19. 2026Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Modelscited by 2
  20. 2026Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLMcited by 2
  21. 2025Speak Your Mind: The Speech Continuation Task as a Probe of Voice-Based Model Biascited by 0
  22. 2026On the Effect of Segmentation Width and Cluster Size on Speech Resynthesis and Continuation in Generative Spoken Language Modelscited by 0
  23. 2026Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Modelscited by 0
  24. 2026Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
  2. 2026Quantifying the Generation Modality Gap in Speech-Text Language Models
  3. 2026Raon-OpenTTS: Open Models and Data for Robust Text-to-Speech ↗
  4. 2026MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching
  5. 2025ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching
  6. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias
  7. 2024Instruction Data Generation and Unsupervised Adaptation for Speech Language Models
  8. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis