# Cycle-consistency training for end-to-end speech recognition

arXiv: https://arxiv.org/abs/1811.01690 · HTML: https://mlsanity.com/paper/1811.01690/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2018.

## Built on by (most cited first, 30 of 31)

- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition](https://mlsanity.com/paper/2106.08922/) (2021)
- [Injecting Text in Self-Supervised Speech Pretraining](https://mlsanity.com/paper/2108.12226/) (2021)
- [fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit](https://mlsanity.com/paper/2109.06912/) (2021)
- [Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognition](https://mlsanity.com/paper/2110.11479/) (2021)
- [JOIST: A Joint Speech and Text Streaming Model For ASR](https://mlsanity.com/paper/2210.07353/) (2022)
- [OkwuGb'e: End-to-End Speech Recognition for Fon and Igbo](https://mlsanity.com/paper/2103.07762/) (2021)
- [EAT: Enhanced ASR-TTS for Self-supervised Speech Recognition](https://mlsanity.com/paper/2104.07474/) (2021)
- [Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generator](https://mlsanity.com/paper/2302.14036/) (2023)
- [Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR](https://mlsanity.com/paper/2210.05793/) (2022)
- [Massively Multilingual Shallow Fusion with Large Language Models](https://mlsanity.com/paper/2302.08917/) (2023)
- [UALM: Unified Audio Language Model for Understanding, Generation and Reasoning](https://mlsanity.com/paper/2510.12000/) (2025)
- [Generating Data with Text-to-Speech and Large-Language Models for Conversational Speech Recognition](https://mlsanity.com/paper/2408.09215/) (2024)
- [Make More of Your Data: Minimal Effort Data Augmentation for Automatic Speech Recognition and Translation](https://mlsanity.com/paper/2210.15398/) (2022)
- [Text Generation with Speech Synthesis for ASR Data Augmentation](https://mlsanity.com/paper/2305.16333/) (2023)
- [Speech-text based multi-modal training with bidirectional attention for improved speech recognition](https://mlsanity.com/paper/2211.00325/) (2022)
- [Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuning](https://mlsanity.com/paper/2412.06967/) (2024)
- [Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions](https://mlsanity.com/paper/2602.05220/) (2026)
- [End-to-End Rich Transcription-Style Automatic Speech Recognition with Semi-Supervised Learning](https://mlsanity.com/paper/2107.05382/) (2021)
- [Learning the joint distribution of two sequences using little or no paired data](https://mlsanity.com/paper/2212.03232/) (2022)
- [SpeechComposer: Unifying Multiple Speech Tasks with Prompt Composition](https://mlsanity.com/paper/2401.18045/) (2024)
- [Augmenting text for spoken language understanding with Large Language Models](https://mlsanity.com/paper/2309.09390/) (2023)
- [Improving noisy student training for low-resource languages in End-to-End ASR using CycleGAN and inter-domain losses](https://mlsanity.com/paper/2407.21061/) (2024)
- [A Complementary Joint Training Approach Using Unpaired Speech and Text for Low-Resource Automatic Speech Recognition](https://mlsanity.com/paper/2204.02023/) (2022)
- [Improving Semi-supervised End-to-end Automatic Speech Recognition using CycleGAN and Inter-domain Losses](https://mlsanity.com/paper/2210.11642/) (2022)
- [A Deliberation-based Joint Acoustic and Text Decoder](https://mlsanity.com/paper/2303.15293/) (2023)
- [A Comparison of Semi-Supervised Learning Techniques for Streaming ASR at Scale](https://mlsanity.com/paper/2304.11053/) (2023)
- [Dual Learning for Large Vocabulary On-Device ASR](https://mlsanity.com/paper/2301.04327/) (2023)

## Abstract (arXiv)

This paper presents a method to train end-to-end automatic speech recognition (ASR) models using unpaired data. Although the end-to-end approach can eliminate the need for expert knowledge such as pronunciation dictionaries to build ASR systems, it still requires a large amount of paired data, i.e., speech utterances and their transcriptions. Cycle-consistency losses have been recently proposed as a way to mitigate the problem of limited paired data. These approaches compose a reverse operation with a given transformation, e.g., text-to-speech (TTS) with ASR, to build a loss that only requires unsupervised data, speech in this example. Applying cycle consistency to ASR models is not trivial since fundamental information, such as speaker traits, are lost in the intermediate text bottleneck. To solve this problem, this work presents a loss that is based on the speech encoder state sequence instead of the raw speech signal. This is achieved by training a Text-To-Encoder model and defining a loss based on the encoder reconstruction error. Experimental results on the LibriSpeech corpus show that the proposed cycle-consistency training reduced the word error rate by 14.7% from an initial model trained with 100-hour paired data, using an additional 360 hours of audio data without transcriptions. We also investigate the use of text-only data mainly for language modeling to further improve the performance in the unpaired data training scenario.