# Hierarchical Multitask Learning for CTC-based Speech Recognition

arXiv: https://arxiv.org/abs/1807.06234 · HTML: https://mlsanity.com/paper/1807.06234/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2018.

## Built on by (most cited first, 14 of 14)

- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021)
- [Leveraging Hierarchical Structures for Few-Shot Musical Instrument Recognition](https://mlsanity.com/paper/2107.07029/) (2021)
- [Hierarchical Conditional End-to-End ASR with CTC and Multi-Granular Subword Units](https://mlsanity.com/paper/2110.04109/) (2021)
- [Non-autoregressive Error Correction for CTC-based ASR with Phone-conditioned Masked LM](https://mlsanity.com/paper/2209.04062/) (2022)
- [Better Intermediates Improve CTC Inference](https://mlsanity.com/paper/2204.00176/) (2022)
- [Improvement in Sign Language Translation Using Text CTC Alignment](https://mlsanity.com/paper/2412.09014/) (2024)
- [K-Wav2vec 2.0: Automatic Speech Recognition based on Joint Decoding of Graphemes and Syllables](https://mlsanity.com/paper/2110.05172/) (2021)
- [Improving CTC-based ASR Models with Gated Interlayer Collaboration](https://mlsanity.com/paper/2205.12462/) (2022)
- [PM-MMUT: Boosted Phone-Mask Data Augmentation using Multi-Modeling Unit Training for Phonetic-Reduction-Robust E2E Speech Recognition](https://mlsanity.com/paper/2112.06721/) (2021)
- [Dynamic Restrained Uncertainty Weighting Loss for Multitask Learning of Vocal Expression](https://mlsanity.com/paper/2206.11049/) (2022)
- [Knowledge distillation from language model to acoustic model: a hierarchical multi-task learning approach](https://mlsanity.com/paper/2110.10429/) (2021)
- [Transfer Learning in Conversational Analysis through Reusing Preprocessing Data as Supervisors](https://mlsanity.com/paper/2112.03032/) (2021)
- [Multi-Level Modeling Units for End-to-End Mandarin Speech Recognition](https://mlsanity.com/paper/2205.11998/) (2022)

## Abstract (arXiv)

Previous work has shown that neural encoder-decoder speech recognition can be improved with hierarchical multitask learning, where auxiliary tasks are added at intermediate layers of a deep encoder. We explore the effect of hierarchical multitask learning in the context of connectionist temporal classification (CTC)-based speech recognition, and investigate several aspects of this approach. Consistent with previous work, we observe performance improvements on telephone conversational speech recognition (specifically the Eval2000 test sets) when training a subword-level CTC model with an auxiliary phone loss at an intermediate layer. We analyze the effects of a number of experimental variables (like interpolation constant and position of the auxiliary loss function), performance in lower-resource settings, and the relationship between pretraining and multitask learning. We observe that the hierarchical multitask approach improves over standard multitask training in our higher-data experiments, while in the low-resource settings standard multitask training works well. The best results are obtained by combining hierarchical multitask learning and pretraining, which improves word error rates by 3.4% absolute on the Eval2000 test sets.