# Adversarial Training of End-to-end Speech Recognition Using a Criticizing Language Model

arXiv: https://arxiv.org/abs/1811.00787 · HTML: https://mlsanity.com/paper/1811.00787/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2018.

## Built on by (most cited first, 12 of 12)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Injecting Text in Self-Supervised Speech Pretraining](https://mlsanity.com/paper/2108.12226/) (2021)
- [Efficiently Fusing Pretrained Acoustic and Linguistic Encoders for Low-resource Speech Recognition](https://mlsanity.com/paper/2101.06699/) (2021)
- [SpeechNet: A Universal Modularized Model for Speech Processing Tasks](https://mlsanity.com/paper/2105.03070/) (2021)
- [Understanding and Improvement of Adversarial Training for Network Embedding from an Optimization Perspective](https://mlsanity.com/paper/2105.08007/) (2021)
- [Fine-tuning of Pre-trained End-to-end Speech Recognition with Generative Adversarial Networks](https://mlsanity.com/paper/2103.13329/) (2021)
- [CoLM-DSR: Leveraging Neural Codec Language Modeling for Multi-Modal Dysarthric Speech Reconstruction](https://mlsanity.com/paper/2406.08336/) (2024)
- [Injecting Text and Cross-lingual Supervision in Few-shot Learning from Self-Supervised Models](https://mlsanity.com/paper/2110.04863/) (2021)
- [Adaptive Sparse and Monotonic Attention for Transformer-based Automatic Speech Recognition](https://mlsanity.com/paper/2209.15176/) (2022)
- [Unified Signal Compression Using a GAN with Iterative Latent Representation Optimization](https://mlsanity.com/paper/2109.11168/) (2021)
- [Improved Self-Supervised Multilingual Speech Representation Learning Combined with Auxiliary Language Information](https://mlsanity.com/paper/2212.03476/) (2022)
- [Exploring Turkish Speech Recognition via Hybrid CTC/Attention Architecture and Multi-feature Fusion Network](https://mlsanity.com/paper/2303.12300/) (2023)

## Abstract (arXiv)

In this paper we proposed a novel Adversarial Training (AT) approach for end-to-end speech recognition using a Criticizing Language Model (CLM). In this way the CLM and the automatic speech recognition (ASR) model can challenge and learn from each other iteratively to improve the performance. Since the CLM only takes the text as input, huge quantities of unpaired text data can be utilized in this approach within end-to-end training. Moreover, AT can be applied to any end-to-end ASR model using any deep-learning-based language modeling frameworks, and compatible with any existing end-to-end decoding method. Initial results with an example experimental setup demonstrated the proposed approach is able to gain consistent improvements efficiently from auxiliary text data under different scenarios.