# Improving Frame-level Classifier for Word Timings with Non-peaky CTC in End-to-End Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2306.07949 · HTML: https://mlsanity.com/paper/2306.07949/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 9 Jun 2023.

## Builds on (most cited first, 15 of 15)

- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling](https://mlsanity.com/paper/1902.08295/) (2019)
- [EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding](https://mlsanity.com/paper/1507.08240/) (2015)
- [Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01818/) (2017)
- [Why does CTC result in peaky behavior?](https://mlsanity.com/paper/2105.14849/) (2021)
- [RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition](https://mlsanity.com/paper/1805.05225/) (2018)
- [Achieving Timestamp Prediction While Recognizing with Non-Autoregressive End-to-End ASR Model](https://mlsanity.com/paper/2301.12343/) (2023)
- [Peak-First CTC: Reducing the Peak Latency of CTC Models by Applying Peak-First Regularization](https://mlsanity.com/paper/2211.03284/) (2022)
- [HMM-Free Encoder Pre-Training for Streaming RNN Transducer](https://mlsanity.com/paper/2104.10764/) (2021)

## Built on by (most cited first, 1 of 1)

- [End-to-End Training of a Neural HMM with Label and Transition Probabilities](https://mlsanity.com/paper/2310.02724/) (2023)

## Abstract (arXiv)

End-to-end (E2E) systems have shown comparable performance to hybrid systems for automatic speech recognition (ASR). Word timings, as a by-product of ASR, are essential in many applications, especially for subtitling and computer-aided pronunciation training. In this paper, we improve the frame-level classifier for word timings in E2E system by introducing label priors in connectionist temporal classification (CTC) loss, which is adopted from prior works, and combining low-level Mel-scale filter banks with high-level ASR encoder output as input feature. On the internal Chinese corpus, the proposed method achieves 95.68%/94.18% compared to the hybrid system 93.0%/90.22% on the word timing accuracy metrics. It also surpass a previous E2E approach with an absolute increase of 4.80%/8.02% on the metrics on 7 languages. In addition, we further improve word timing accuracy by delaying CTC peaks with frame-wise knowledge distillation, though only experimenting on LibriSpeech.