# Alternating Weak Triphone/BPE Alignment Supervision from Hybrid Model Improves End-to-End ASR

arXiv: https://arxiv.org/abs/2402.15594 · HTML: https://mlsanity.com/paper/2402.15594/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 23 Feb 2024.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021)
- [Multi-task Sequence to Sequence Learning](https://mlsanity.com/paper/1511.06114/) (2015)
- [E-Branchformer: Branchformer with Enhanced merging for speech recognition](https://mlsanity.com/paper/2210.00077/) (2022)
- [Alignment Restricted Streaming Recurrent Neural Network Transducer](https://mlsanity.com/paper/2011.03072/) (2020)
- [Citrinet: Closing the Gap between Non-Autoregressive and Autoregressive End-to-End Models for Automatic Speech Recognition](https://mlsanity.com/paper/2104.01721/) (2021)
- [RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition](https://mlsanity.com/paper/1805.05225/) (2018)
- [Multitask Learning with Low-Level Auxiliary Tasks for Encoder-Decoder Based Speech Recognition](https://mlsanity.com/paper/1704.01631/) (2017)
- [Improving RNN Transducer Based ASR with Auxiliary Tasks](https://mlsanity.com/paper/2011.03109/) (2020)
- [RNN-T For Latency Controlled ASR With Improved Beam Search](https://mlsanity.com/paper/1911.01629/) (2019)
- [Guiding CTC Posterior Spike Timings for Improved Posterior Fusion and Knowledge Distillation](https://mlsanity.com/paper/1904.08311/) (2019)
- [A Study of Transducer based End-to-End ASR with ESPnet: Architecture, Auxiliary Loss and Decoding Strategies](https://mlsanity.com/paper/2201.05420/) (2022)
- [Revisiting Checkpoint Averaging for Neural Machine Translation](https://mlsanity.com/paper/2210.11803/) (2022)
- [Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognition](https://mlsanity.com/paper/2309.08436/) (2023)
- [Weak Alignment Supervision from Hybrid Model Improves End-to-end ASR](https://mlsanity.com/paper/2311.14835/) (2023)

## Abstract (arXiv)

In this paper, alternating weak triphone/BPE alignment supervision is proposed to improve end-to-end model training. Towards this end, triphone and BPE alignments are extracted using a pre-existing hybrid ASR system. Then, regularization effect is obtained by cross-entropy based intermediate auxiliary losses computed on such alignments at a mid-layer representation of the encoder for triphone alignments and at the encoder for BPE alignments. Weak supervision is achieved through strong label smoothing with parameter of 0.5. Experimental results on TED-LIUM 2 indicate that either triphone or BPE alignment based weak supervision improves ASR performance over standard CTC auxiliary loss. Moreover, their combination lowers the word error rate further. We also investigate the alternation of the two auxiliary tasks during model training, and additional performance gain is observed. Overall, the proposed techniques result in over 10% relative error rate reduction over a CTC-regularized baseline system.