A Non-autoregressive Model for Joint STT and TTS
Not on the map. Why?
Abstract (arXiv)
In this paper, we take a step towards jointly modeling automatic speech recognition (STT) and speech synthesis (TTS) in a fully non-autoregressive way. We develop a novel multimodal framework capable of handling the speech and text modalities as input either individually or together. The proposed model can also be trained with unpaired speech or text data owing to its multimodal nature. We further propose an iterative refinement strategy to improve the STT and TTS performance of our model such that the partial hypothesis at the output can be fed back to the input of our model, thus iteratively improving both STT and TTS predictions. We show that our joint model can effectively perform both STT and TTS tasks, outperforming the STT-specific baseline in all tasks and performing competitively with the TTS-specific baseline across a wide range of evaluation metrics.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 15 Jan 2025.
Builds on
27 of 38 references.
References · 27
- 2016Layer Normalizationcited by 5,579
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
- 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
- 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
- 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
- 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 124
15 more
- 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 118
- 2023LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpuscited by 100
- 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 77
- 2020Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predictcited by 75
- 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 62
- 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
- 2019Almost Unsupervised Text to Speech and Automatic Speech Recognitioncited by 41
- 2020Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignmentcited by 37
- 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
- 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
- 2024ESPnet-SPK: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf modelscited by 22
- 2022Integrating Text Inputs For Training and Adapting RNN Transducer ASR Modelscited by 20
- 2022BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Modelcited by 11
- 2022BECTRA: Transducer-based End-to-End ASR with BERT-Enhanced Encodercited by 8
- 2023Semi-Autoregressive Streaming ASR With Label Contextcited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.