# Semantic Mask for Transformer based End-to-End Speech Recognition

arXiv: https://arxiv.org/abs/1912.03010 · HTML: https://mlsanity.com/paper/1912.03010/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2019.

## Built on by (most cited first, 18 of 18)

- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [E-Branchformer: Branchformer with Enhanced merging for speech recognition](https://mlsanity.com/paper/2210.00077/) (2022)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [MixSpeech: Data Augmentation for Low-resource Automatic Speech Recognition](https://mlsanity.com/paper/2102.12664/) (2021)
- [Streaming Multi-Talker ASR with Token-Level Serialized Output Training](https://mlsanity.com/paper/2202.00842/) (2022)
- [Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition](https://mlsanity.com/paper/2103.00422/) (2021)
- [SpliceOut: A Simple and Efficient Audio Augmentation Method](https://mlsanity.com/paper/2110.00046/) (2021)
- [Fine-tuning of Pre-trained End-to-end Speech Recognition with Generative Adversarial Networks](https://mlsanity.com/paper/2103.13329/) (2021)
- [A Survey on Dropout Methods and Experimental Verification in Recommendation](https://mlsanity.com/paper/2204.02027/) (2022)
- [Transformer-based ASR Incorporating Time-reduction Layer and Fine-tuning with Self-Knowledge Distillation](https://mlsanity.com/paper/2103.09903/) (2021)
- [PM-MMUT: Boosted Phone-Mask Data Augmentation using Multi-Modeling Unit Training for Phonetic-Reduction-Robust E2E Speech Recognition](https://mlsanity.com/paper/2112.06721/) (2021)
- [Ask2Mask: Guided Data Selection for Masked Speech Modeling](https://mlsanity.com/paper/2202.12719/) (2022)
- [Ultra Fast Speech Separation Model with Teacher Student Learning](https://mlsanity.com/paper/2204.12777/) (2022)
- [Semi-Supervised Learning Based on Reference Model for Low-resource TTS](https://mlsanity.com/paper/2210.14723/) (2022)
- [Regularizing Learnable Feature Extraction for Automatic Speech Recognition](https://mlsanity.com/paper/2506.09804/) (2025)
- [SCaLa: Supervised Contrastive Learning for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.04187/) (2021)

## Abstract (arXiv)

Attention-based encoder-decoder model has achieved impressive results for both automatic speech recognition (ASR) and text-to-speech (TTS) tasks. This approach takes advantage of the memorization capacity of neural networks to learn the mapping from the input sequence to the output sequence from scratch, without the assumption of prior knowledge such as the alignments. However, this model is prone to overfitting, especially when the amount of training data is limited. Inspired by SpecAugment and BERT, in this paper, we propose a semantic mask based regularization for training such kind of end-to-end (E2E) model. The idea is to mask the input features corresponding to a particular output token, e.g., a word or a word-piece, in order to encourage the model to fill the token based on the contextual information. While this approach is applicable to the encoder-decoder framework with any type of neural network architecture, we study the transformer-based model for ASR in this work. We perform experiments on Librispeech 960h and TedLium2 data sets, and achieve the state-of-the-art performance on the test set in the scope of E2E models.