# EAT: Enhanced ASR-TTS for Self-supervised Speech Recognition

arXiv: https://arxiv.org/abs/2104.07474 · HTML: https://mlsanity.com/paper/2104.07474/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 13 Apr 2021.

## Builds on (most cited first, 18 of 18)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Self-Training for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.09116/) (2019)
- [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://mlsanity.com/paper/1911.08460/) (2019)
- [Effectiveness of self-supervised pre-training for speech recognition](https://mlsanity.com/paper/1911.03912/) (2019)
- [ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit](https://mlsanity.com/paper/1910.10909/) (2019)
- [Generating Synthetic Audio Data for Attention-Based Speech Recognition Systems](https://mlsanity.com/paper/1912.09257/) (2019)
- [Listening while Speaking: Speech Chain by Deep Learning](https://mlsanity.com/paper/1707.04879/) (2017)
- [Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and auto-regressive prosody prior](https://mlsanity.com/paper/2002.03788/) (2020)
- [Cycle-consistency training for end-to-end speech recognition](https://mlsanity.com/paper/1811.01690/) (2018)
- [Back-Translation-Style Data Augmentation for End-to-End ASR](https://mlsanity.com/paper/1807.10893/) (2018)
- [Semi-Supervised Speech Recognition via Local Prior Matching](https://mlsanity.com/paper/2002.10336/) (2020)
- [End-to-End Feedback Loss in Speech Chain Framework via Straight-Through Estimator](https://mlsanity.com/paper/1810.13107/) (2018)
- [Semi-supervised Sequence-to-sequence ASR using Unpaired Speech and Text](https://mlsanity.com/paper/1905.01152/) (2019)

## Built on by (most cited first, 13 of 13)

- [Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognition](https://mlsanity.com/paper/2110.11479/) (2021)
- [Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generator](https://mlsanity.com/paper/2302.14036/) (2023)
- [Boosting Cross-Domain Speech Recognition with Self-Supervision](https://mlsanity.com/paper/2206.09783/) (2022)
- [Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognition](https://mlsanity.com/paper/2408.09215/) (2024)
- [USTED: Improving ASR with a Unified Speech and Text Encoder-Decoder](https://mlsanity.com/paper/2202.06045/) (2022)
- [Text Generation with Speech Synthesis for ASR Data Augmentation](https://mlsanity.com/paper/2305.16333/) (2023)
- [On the Relevance of Phoneme Duration Variability of Synthesized Training Data for Automatic Speech Recognition](https://mlsanity.com/paper/2310.08132/) (2023)
- [On the Effect of Purely Synthetic Training Data for Different Automatic Speech Recognition Architectures](https://mlsanity.com/paper/2407.17997/) (2024)
- [On the Problem of Text-To-Speech Model Selection for Synthetic Data Generation in Automatic Speech Recognition](https://mlsanity.com/paper/2407.21476/) (2024)
- [SpeeChain: A Speech Toolkit for Large-Scale Machine Speech Chain](https://mlsanity.com/paper/2301.02966/) (2023)
- [Exploring the Viability of Synthetic Audio Data for Audio-Based Dialogue State Tracking](https://mlsanity.com/paper/2312.01842/) (2023)
- [End-to-End Open Vocabulary Keyword Search With Multilingual Neural Representations](https://mlsanity.com/paper/2308.08027/) (2023)
- [Written Term Detection Improves Spoken Term Detection](https://mlsanity.com/paper/2407.04601/) (2024)

## Abstract (arXiv)

Self-supervised ASR-TTS models suffer in out-of-domain data conditions. Here we propose an enhanced ASR-TTS (EAT) model that incorporates two main features: 1) The ASRrightarrowTTS direction is equipped with a language model reward to penalize the ASR hypotheses before forwarding it to TTS. 2) In the TTSrightarrowASR direction, a hyper-parameter is introduced to scale the attention context from synthesized speech before sending it to ASR to handle out-of-domain data. Training strategies and the effectiveness of the EAT model are explored under out-of-domain data conditions. The results show that EAT reduces the performance gap between supervised and self-supervised training significantly by absolute 2.6% and 2.7% on Librispeech and BABEL respectively.