# InterAug: Augmenting Noisy Intermediate Predictions for CTC-based ASR

arXiv: https://arxiv.org/abs/2204.00174 · HTML: https://mlsanity.com/paper/2204.00174/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 1 Apr 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [Dataset Augmentation in Feature Space](https://mlsanity.com/paper/1702.05538/) (2017)
- [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021)
- [Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict](https://mlsanity.com/paper/2005.08700/) (2020)
- [Imputer: Sequence Modelling via Imputation and Dynamic Programming](https://mlsanity.com/paper/2002.08926/) (2020)
- [Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictions](https://mlsanity.com/paper/2104.02724/) (2021)
- [Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignment](https://mlsanity.com/paper/2010.14233/) (2020)
- [Improved Mask-CTC for Non-Autoregressive End-to-End ASR](https://mlsanity.com/paper/2010.13270/) (2020)
- [Spike-Triggered Non-Autoregressive Transformer for End-to-End Speech Recognition](https://mlsanity.com/paper/2005.07903/) (2020)
- [Listen Attentively, and Spell Once: Whole Sentence Generation via a Non-Autoregressive Architecture for Low-Latency Speech Recognition](https://mlsanity.com/paper/2005.04862/) (2020)
- [Insertion-Based Modeling for End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2005.13211/) (2020)
- [SpecAugment++: A Hidden Space Data Augmentation Method for Acoustic Scene Classification](https://mlsanity.com/paper/2103.16858/) (2021)

## Built on by (most cited first, 5 of 5)

- [InterBiasing: Boost Unseen Word Recognition through Biasing Intermediate Predictions](https://mlsanity.com/paper/2406.14890/) (2024)
- [Rapid Language Adaptation for Multilingual E2E Speech Recognition Using Encoder Prompting](https://mlsanity.com/paper/2406.12611/) (2024)
- [Research on an improved Conformer end-to-end Speech Recognition Model with R-Drop Structure](https://mlsanity.com/paper/2306.08329/) (2023)
- [DESign: Dynamic Context-Aware Convolution and Efficient Subnet Regularization for Continuous Sign Language Recognition](https://mlsanity.com/paper/2507.03339/) (2025)
- [Curriculum-Based Noise Adaptation for Phoneme-to-Text Reconstruction in Visual Speech Recognition](https://mlsanity.com/paper/2609.20839/) (2026)

## Abstract (arXiv)

This paper proposes InterAug: a novel training method for CTC-based ASR using augmented intermediate representations for conditioning. The proposed method exploits the conditioning framework of self-conditioned CTC to train robust models by conditioning with "noisy" intermediate predictions. During the training, intermediate predictions are changed to incorrect intermediate predictions, and fed into the next layer for conditioning. The subsequent layers are trained to correct the incorrect intermediate predictions with the intermediate losses. By repeating the augmentation and the correction, iterative refinements, which generally require a special decoder, can be realized only with the audio encoder. To produce noisy intermediate predictions, we also introduce new augmentation: intermediate feature space augmentation and intermediate token space augmentation that are designed to simulate typical errors. The combination of the proposed InterAug framework with new augmentation allows explicit training of the robust audio encoders. In experiments using augmentations simulating deletion, insertion, and substitution error, we confirmed that the trained model acquires robustness to each error, boosting the speech recognition performance of the strong self-conditioned CTC baseline.