# Hierarchical Multi Task Learning With CTC

arXiv: https://arxiv.org/abs/1807.07104 · HTML: https://mlsanity.com/paper/1807.07104/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2018.

## Built on by (most cited first, 20 of 20)

- [Visual Alignment Constraint for Continuous Sign Language Recognition](https://mlsanity.com/paper/2104.02330/) (2021)
- [Improving Massively Multilingual ASR With Auxiliary CTC Objectives](https://mlsanity.com/paper/2302.12829/) (2023)
- [CTC Alignments Improve Autoregressive Translation](https://mlsanity.com/paper/2210.05200/) (2022)
- [Leveraging Hierarchical Structures for Few-Shot Musical Instrument Recognition](https://mlsanity.com/paper/2107.07029/) (2021)
- [Hierarchical Conditional End-to-End ASR with CTC and Multi-Granular Subword Units](https://mlsanity.com/paper/2110.04109/) (2021)
- [BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model](https://mlsanity.com/paper/2210.16663/) (2022)
- [ESPnet-ST-v2: Multipurpose Spoken Language Translation Toolkit](https://mlsanity.com/paper/2304.04596/) (2023)
- [Contextualized End-to-end Automatic Speech Recognition with Intermediate Biasing Loss](https://mlsanity.com/paper/2406.16120/) (2024)
- [End-to-end Speech-to-Punctuated-Text Recognition](https://mlsanity.com/paper/2207.03169/) (2022)
- [Multilingual Gloss-free Sign Language Translation: Towards Building a Sign Language Foundation Model](https://mlsanity.com/paper/2505.24355/) (2025)
- [Better Intermediates Improve CTC Inference](https://mlsanity.com/paper/2204.00176/) (2022)
- [An Empirical Recipe for Universal Phone Recognition](https://mlsanity.com/paper/2603.29042/) (2026)
- [K-Wav2vec 2.0: Automatic Speech Recognition based on Joint Decoding of Graphemes and Syllables](https://mlsanity.com/paper/2110.05172/) (2021)
- [CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processing](https://mlsanity.com/paper/2412.04425/) (2024)
- [Unifying the Discrete and Continuous Emotion labels for Speech Emotion Recognition](https://mlsanity.com/paper/2210.16642/) (2022)
- [Alternate Intermediate Conditioning with Syllable-level and Character-level Targets for Japanese ASR](https://mlsanity.com/paper/2204.00175/) (2022)
- [Multi-Level Modeling Units for End-to-End Mandarin Speech Recognition](https://mlsanity.com/paper/2205.11998/) (2022)
- [InterMPL: Momentum Pseudo-Labeling with Intermediate CTC Loss](https://mlsanity.com/paper/2211.00795/) (2022)
- [Leveraging Language ID to Calculate Intermediate CTC Loss for Enhanced Code-Switching Speech Recognition](https://mlsanity.com/paper/2312.09583/) (2023)
- [Enhancing Code-Switching ASR Leveraging Non-Peaky CTC Loss and Deep Language Posterior Injection](https://mlsanity.com/paper/2412.08651/) (2024)

## Abstract (arXiv)

In Automatic Speech Recognition it is still challenging to learn useful intermediate representations when using high-level (or abstract) target units such as words. For that reason, character or phoneme based systems tend to outperform word-based systems when just few hundreds of hours of training data are being used. In this paper, we first show how hierarchical multi-task training can encourage the formation of useful intermediate representations. We achieve this by performing Connectionist Temporal Classification at different levels of the network with targets of different granularity. Our model thus performs predictions in multiple scales for the same input. On the standard 300h Switchboard training setup, our hierarchical multi-task architecture exhibits improvements over single-task architectures with the same number of parameters. Our model obtains 14.0% Word Error Rate on the Eval2000 Switchboard subset without any decoder or language model, outperforming the current state-of-the-art on acoustic-to-word models.