mlsanityData to 14 Oct 2026

SpeechX: Neural Codec Language Model as a Versatile Speech Transformer

arXiv 2308.06873 ↗First posted 14 Aug 2023eess.AS cs.CL cs.LGcited by 64

Not on the map. Why?

Abstract (arXiv)

Recent advancements in generative speech models based on audio-text prompts have enabled remarkable innovations like high-quality zero-shot text-to-speech. However, existing models still face limitations in handling diverse audio-text speech generation tasks involving transforming input speech and processing audio captured in adverse acoustic conditions. This paper introduces SpeechX, a versatile speech generation model capable of zero-shot TTS and various speech transformation tasks, dealing with both clean and noisy signals. SpeechX combines neural codec language modeling with multi-task learning using task-dependent prompting, enabling unified and extensible modeling and providing a consistent way for leveraging textual input in speech enhancement and transformation tasks. Experimental results show SpeechX's efficacy in various tasks, including zero-shot TTS, noise suppression, target speaker extraction, speech removal, and speech editing with or without background noise, achieving comparable or superior performance to specialized models across tasks. See https://aka.ms/speechx for demo samples.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Builds on 37 of its 43 references that are in the map. First posted on arXiv 14 Aug 2023.

Builds on

37 of 43 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  9. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  10. 2022High Fidelity Neural Audio Compressioncited by 687
  11. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  12. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
25 more
  1. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  2. 2023MusicLM: Generating Music From Textcited by 448
  3. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  4. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  5. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  6. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 282
  7. 2021YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyonecited by 278
  8. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 241
  9. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  10. 2023Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervisioncited by 169
  11. 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
  12. 2023Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesiscited by 153
  13. 2020DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancementcited by 141
  14. 2022Universal Speech Enhancement with Score-based Diffusioncited by 75
  15. 2018VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Maskingcited by 73
  16. 2019Zero-Shot Multi-Speaker Text-To-Speech with State-of-the-art Neural Speaker Embeddingscited by 73
  17. 2023ICASSP 2023 Deep Noise Suppression Challengecited by 65
  18. 2021SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Modelcited by 56
  19. 2023Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Biascited by 51
  20. 2022A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editingcited by 40
  21. 2023Neural Target Speech Extraction: An Overviewcited by 33
  22. 2023Make-A-Voice: Unified Voice Synthesis With Discrete Representationcited by 27
  23. 2021Personalized Speech Enhancement: New Models and Comprehensive Evaluationcited by 21
  24. 2021Human Listening and Live Captioning: Multi-Task Training for Speech Enhancementcited by 8
  25. 2023i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Datacited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 31; 2025: 24; 2026: 6.

64 papers.

Sort:

By area:

  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  2. 2024VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizerscited by 122
  3. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  4. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 113
  5. 2024BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of datacited by 75
  6. 2024ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reorderingcited by 50
  7. 2025Recent Advances in Discrete Speech Tokens: A Reviewcited by 43
  8. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  9. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  10. 2024The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audiocited by 31
  11. 2025Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidancecited by 27
  12. 2024MaskSR: Masked Language Model for Full-band Speech Restorationcited by 24
38 more
  1. 2024How Should We Extract Discrete Audio Tokens from Self-Supervised Models?cited by 24
  2. 2025AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancementcited by 23
  3. 2024Improving Robustness of LLM-based Speech Synthesis by Learning Monotonic Alignmentcited by 22
  4. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  5. 2024SongCreator: Lyrics-based Universal Song Generationcited by 12
  6. 2024Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Likecited by 11
  7. 2025OpusLM: A Family of Open Unified Speech Language Modelscited by 9
  8. 2024Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representationscited by 8
  9. 2023uSee: Unified Speech Enhancement and Editing with Conditional Diffusion Modelscited by 7
  10. 2024StyleTTS-ZS: Efficient High-Quality Zero-Shot Text-to-Speech Synthesis with Distilled Time-Varying Style Diffusioncited by 7
  11. 2025Speech Enhancement Using Continuous Embeddings of Neural Audio Codeccited by 7
  12. 2025Universal Speech Enhancement with Regression and Generative Mambacited by 7
  13. 2025Multi-Metric Preference Alignment for Generative Speech Restorationcited by 7
  14. 2023Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extractioncited by 6
  15. 2024Codecfake: An Initial Dataset for Detecting LLM-based Deepfake Audiocited by 6
  16. 2025FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusioncited by 6
  17. 2025UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalitiescited by 6
  18. 2024Gull: A Generative Multifunctional Audio Codeccited by 5
  19. 2024A Survey of Deep Learning Audio Generation Methodscited by 5
  20. 2024Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibilitycited by 5
  21. 2024DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesiscited by 5
  22. 2025UniWav: Towards Unified Pre-training for Speech Representation Learning and Generationcited by 5
  23. 2025LauraTSE: Target Speaker Extraction using Auto-Regressive Decoder-Only Language Modelscited by 5
  24. 2024An Investigation of Noise Robustness for Flow-Matching-Based Zero-Shot TTScited by 4
  25. 2024Towards Audio Codec-based Speech Separationcited by 4
  26. 2025Voice Cloning: Comprehensive Surveycited by 4
  27. 2026Bagpiper: Solving Open-Ended Audio Tasks via Rich Captionscited by 4
  28. 2024SpeechComposer: Unifying Multiple Speech Tasks with Prompt Compositioncited by 3
  29. 2024FViT: A Focal Vision Transformer with Gabor Filtercited by 3
  30. 2024Bailing-TTS: Chinese Dialectal Speech Synthesis Towards Human-like Spontaneous Representationcited by 3
  31. 2024Continuous Speech Tokenizer in Text To Speechcited by 3
  32. 2024A Closer Look at Neural Codec Resynthesis: Bridging the Gap between Codec and Waveform Generationcited by 3
  33. 2025VoicePrompter: Robust Zero-Shot Voice Conversion with Voice Prompt and Conditional Flow Matchingcited by 3
  34. 2025Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecscited by 3
  35. 2024Does Current Deepfake Audio Detection Model Effectively Detect ALM-based Deepfake Audio?cited by 2
  36. 2024Text-to-Speech for Unseen Speakers via Low-Complexity Discrete Unit-Based Frame Selectioncited by 2
  37. 2024An Empirical Study of Speech Language Models for Prompt-Conditioned Speech Synthesiscited by 1
  38. 2024Speech Editing -- a Summarycited by 1

50 most cited of 64 All 64 (JSON)