mlsanityData to 14 Oct 2026

SpeechOp: Inference-Time Task Composition for Generative Speech Processing

arXiv 2509.14298 ↗First posted 17 Sep 2025eess.AS cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

While generative Text-to-Speech (TTS) systems leverage vast ``in-the-wild" data to achieve remarkable success, speech-to-speech processing tasks like enhancement face data limitations, which lead data-hungry generative approaches to distort speech content and speaker identity. To bridge this gap, we present SpeechOp, a multi-task latent diffusion model that transforms pre-trained TTS models into a universal speech processor capable of performing a wide range of speech tasks and composing them in novel ways at inference time. By adapting a pre-trained TTS model, SpeechOp inherits a rich understanding of natural speech, accelerating training and improving S2S task quality, while simultaneously enhancing core TTS performance. Finally, we introduce Implicit Task Composition (ITC), a novel pipeline where ASR-derived transcripts (e.g., from Whisper) guide SpeechOp's enhancement via our principled inference-time task composition. ITC achieves state-of-the-art content preservation by robustly combining web-scale speech understanding with SpeechOp's generative capabilities. Audio samples are available at https://justinlovelace.github.io/projects/speechop

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 44 of its 54 references that are in the map. First posted on arXiv 17 Sep 2025.

Builds on

44 of 54 references.

References · 44

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2022Scalable Diffusion Models with Transformerscited by 6,054
  4. 2022Classifier-Free Diffusion Guidancecited by 5,847
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  10. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  11. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  12. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
32 more
  1. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  2. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  3. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  4. 2018SDR - half-baked or well done?cited by 423
  5. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 422
  6. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  7. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  8. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
  9. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentationcited by 304
  10. 2024Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Modelscited by 254
  11. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  12. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  13. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Modelscited by 197
  14. 2020Attention is All You Need in Speech Separationcited by 185
  15. 2024XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Modelcited by 179
  16. 2023UniAudio: An Audio Foundation Model Toward Universal Audio Generationcited by 127
  17. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  18. 2022StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberationcited by 105
  19. 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 101
  20. 2024Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusioncited by 75
  21. 2023ICASSP 2023 Deep Noise Suppression Challengecited by 65
  22. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  23. 2019WHAMR!: Noisy and Reverberant Single-Channel Speech Separationcited by 61
  24. 2023Generative Pre-training for Speech with Flow Matchingcited by 55
  25. 2024SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metricscited by 38
  26. 2024CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speechcited by 37
  27. 2023Miipher: A Robust Speech Restoration Model Integrating Self-Supervised Speech and Text Representationscited by 22
  28. 2021Structure from Silence: Learning Scene Structure from Ambient Soundcited by 17
  29. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  30. 2023TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognitioncited by 8
  31. 2024Sample-Efficient Diffusion for Text-To-Speech Synthesiscited by 4
  32. 2024Improving Generalization of Speech Separation in Real-World Scenarios: Strategies in Simulation, Optimization, and Evaluationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025UniFlow: Unifying Speech Front-End Tasks via Continuous Generative Modeling
  2. 2026LongCat-AudioDiT: High-Fidelity Diffusion Text-to-Speech in the Waveform Latent Space
  3. 2025Metis: A Foundation Speech Generation Model with Masked Generative Pre-training
  4. 2026LLaDA-TTS: Unifying Speech Synthesis and Zero-Shot Editing via Masked Diffusion Modeling
  5. 2024On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models
  6. 2026Training a Compact, Strong, and Realistic Text-to-Speech Model
  7. 2025TISDiSS: A Training-Time and Inference-Time Scalable Framework for Discriminative Source Separation
  8. 2025MELA-TTS: Joint transformer-diffusion model with representation alignment for speech synthesis