mlsanityData to 7 Oct 2026

SpeechNet: A Universal Modularized Model for Speech Processing Tasks

arXiv 2105.03070 ↗First posted 7 May 2021cs.CL cs.LG cs.SDcited by 16

Not on the map. Why?

Abstract (arXiv)

There is a wide variety of speech processing tasks ranging from extracting content information from speech signals to generating speech signals. For different tasks, model networks are usually designed and tuned separately. If a universal model can perform multiple speech processing tasks, some tasks might be improved with the related abilities learned from other tasks. The multi-task learning of a wide variety of speech processing tasks with a universal model has not been studied. This paper proposes a universal modularized model, SpeechNet, which treats all speech processing tasks into a speech/text input and speech/text output format. We select five essential speech processing tasks for multi-task learning experiments with SpeechNet. We show that SpeechNet learns all of the above tasks, and we further analyze which tasks can be improved by other tasks. SpeechNet is modularized and flexible for incorporating more modules, tasks, or training approaches in the future. We release the code and experimental settings to facilitate the research of modularized universal models and multi-task learning of speech processing tasks.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 43 of its 80 references that are in the map. First posted on arXiv 7 May 2021.

Builds on

43 of 80 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2017Decoupled Weight Decay Regularizationcited by 1,722
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  11. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  12. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
31 more
  1. 2017A Survey on Multi-Task Learningcited by 838
  2. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  3. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  4. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  5. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  6. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  7. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  8. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  9. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  10. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  11. 2019AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Losscited by 189
  12. 2017One Model To Learn Them Allcited by 86
  13. 2017Deep Voice 2: Multi-Speaker Neural Text-to-Speechcited by 75
  14. 2018The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methodscited by 68
  15. 2017Listening while Speaking: Speech Chain by Deep Learningcited by 46
  16. 2019Almost Unsupervised Text to Speech and Automatic Speech Recognitioncited by 41
  17. 2020FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attentioncited by 37
  18. 2019T-GSA: Transformer with Gaussian-weighted self-attention for speech enhancementcited by 34
  19. 2020Self-attention encoding and pooling for speaker recognitioncited by 33
  20. 2019Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using Transformer with Text-to-Speech Pretrainingcited by 29
  21. 2020Multi-Task Learning for Dense Prediction Tasks: A Surveycited by 28
  22. 2019Non-Parallel Sequence-to-Sequence Voice Conversion with Disentangled Linguistic and Speaker Representationscited by 23
  23. 2020AlignTTS: Efficient Feed-Forward Text-to-Speech System without Explicit Alignmentcited by 20
  24. 2019Joint training framework for text-to-speech and voice conversion using multi-source Tacotron and WaveNetcited by 15
  25. 2018Adversarial Training of End-to-end Speech Recognition Using a Criticizing Language Modelcited by 11
  26. 2019AIPNet: Generative Adversarial Pre-training of Accent-invariant Networks for End-to-end Speech Recognitioncited by 11
  27. 2018Improving Sequence-to-Sequence Acoustic Modeling by Adding Text-Supervisioncited by 10
  28. 2019Bootstrapping non-parallel voice conversion from speaker-adaptive text-to-speechcited by 3
  29. 2020Boosting Objective Scores of a Speech Enhancement Model by MetricGAN Post-processingcited by 3
  30. 2016Multi-task Recurrent Model for Speech and Speaker Recognitioncited by 2
  31. 2020T-vectors: Weakly Supervised Speaker Identification Using Hierarchical Transformer Modelcited by 2

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 2; 2023: 6; 2024: 5; 2025: 1.

16 papers.

Sort:

By area:

  1. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  2. 2024Qwen2-Audio Technical Reportcited by 543
  3. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  4. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  5. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
  6. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 62
  7. 2024Distilling an End-to-End Voice Assistant Without Instruction Training Datacited by 35
  8. 2022Meta Learning for Natural Language Processing: A Surveycited by 26
  9. 2023SpeechGen: Unlocking the Generative Power of Speech Language Models with Promptscited by 21
  10. 2022Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processingcited by 7
  11. 2023Cocktail HuBERT: Generalized Self-Supervised Pre-training for Mixture and Single-Source Speechcited by 6
  12. 2024A Survey of Deep Learning Audio Generation Methodscited by 5
4 more
  1. 2021Speech Representation Learning Through Self-supervised Pretraining And Multi-task Finetuningcited by 4
  2. 2023Spatial HuBERT: Self-supervised Spatial Speech Representation Learning for a Single Talker from Multi-channel Audiocited by 2
  3. 2025UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Modelscited by 1
  4. 2024An Adapter-Based Unified Model for Multiple Spoken Language Processing Taskscited by 0