mlsanityData to 7 Oct 2026

Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generator

arXiv 2302.14036 ↗First posted 27 Feb 2023eess.AS cs.CL cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

We propose an end-to-end Automatic Speech Recognition (ASR) system that can be trained on transcribed speech data, text-only data, or a mixture of both. The proposed model uses an integrated auxiliary block for text-based training. This block combines a non-autoregressive multi-speaker text-to-mel-spectrogram generator with a GAN-based enhancer to improve the spectrogram quality. The proposed system can generate a mel-spectrogram dynamically during training. It can be used to adapt the ASR model to a new domain by using text-only data from this domain. We demonstrate that the proposed training method significantly improves ASR accuracy compared to the system trained on transcribed speech only. It also surpasses cascade TTS systems with the vocoder in the adaptation quality and training speed.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

30 of 42 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Layer Normalizationcited by 5,579
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  9. 2018Which Training Methods for GANs do actually Converge?cited by 672
  10. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 639
  11. 2017Geometric GANcited by 232
  12. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
18 more
  1. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  2. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 150
  3. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  4. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  5. 2021UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generationcited by 71
  6. 2020Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capabilitycited by 63
  7. 2020Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systemscited by 41
  8. 2018Cycle-consistency training for end-to-end speech recognitioncited by 31
  9. 2020You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentationcited by 29
  10. 2021Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognitioncited by 24
  11. 2018Training Neural Speech Recognition Systems with Synthetic Speech Augmentationcited by 21
  12. 2022Integrating Text Inputs For Training and Adapting RNN Transducer ASR Modelscited by 20
  13. 2021CTC Variations Through New WFST Topologiescited by 17
  14. 2022Modular Hybrid Autoregressive Transducercited by 17
  15. 2021EAT: Enhanced ASR-TTS for Self-supervised Speech Recognitioncited by 13
  16. 2021Speaker Generationcited by 13
  17. 2022When Is TTS Augmentation Through a Pivot Language Useful?cited by 7
  18. 2019Independent language modeling architecture for end-to-end ASRcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 5; 2025: 2.

11 papers.

  1. 2024Task Arithmetic can Mitigate Synthetic-to-Real Gap in Automatic Speech Recognitioncited by 14
  2. 2025A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Datacited by 7
  3. 2023Text-only Domain Adaptation using Unified Speech-Text Representation in Transducercited by 5
  4. 2023Corpus Synthesis for Zero-shot ASR domain Adaptation using Large Language Modelscited by 5
  5. 2024Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuningcited by 4
  6. 2023Text Injection for Capitalization and Turn-Taking Prediction in Speech Modelscited by 1
  7. 2024Hard-Synth: Synthesizing Diverse Hard Samples for ASR using Zero-Shot TTS and LLMcited by 1
  8. 2023Wiki-En-ASR-Adapt: Large-scale synthetic dataset for English ASR Customizationcited by 0
  9. 2024Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASRcited by 0
  10. 2024AMPS: ASR with Multimodal Paraphrase Supervisioncited by 0
  11. 2025WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Small Pretrained Speech Recognition Transformerscited by 0