# Longer is (Not Necessarily) Stronger: Punctuated Long-Sequence Training for Enhanced Speech Recognition and Translation

arXiv: https://arxiv.org/abs/2409.05601 · HTML: https://mlsanity.com/paper/2409.05601/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 9 Sep 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio](https://mlsanity.com/paper/2106.06909/) (2021)
- [TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptation](https://mlsanity.com/paper/1805.04699/) (2018)
- [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020)
- [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://mlsanity.com/paper/2305.05084/) (2023)
- [SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognition](https://mlsanity.com/paper/2104.02014/) (2021)
- [Earnings-21: A Practical Benchmark for ASR in the Wild](https://mlsanity.com/paper/2104.11348/) (2021)
- [Training-Free Long-Context Scaling of Large Language Models](https://mlsanity.com/paper/2402.17463/) (2024)
- [Efficient Sequence Transduction by Jointly Predicting Tokens and Durations](https://mlsanity.com/paper/2304.06795/) (2023)
- [Earnings-22: A Practical Benchmark for Accents in the Wild](https://mlsanity.com/paper/2203.15591/) (2022)
- [LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Models](https://mlsanity.com/paper/2310.02943/) (2023)
- [CTC Alignments Improve Autoregressive Translation](https://mlsanity.com/paper/2210.05200/) (2022)
- [Punctuation Prediction Model for Conversational Speech](https://mlsanity.com/paper/1807.00543/) (2018)
- [Investigating End-to-End ASR Architectures for Long Form Audio Transcription](https://mlsanity.com/paper/2309.09950/) (2023)
- [Updated Corpora and Benchmarks for Long-Form Speech Recognition](https://mlsanity.com/paper/2309.15013/) (2023)
- [Joint prediction of truecasing and punctuation for conversational speech in low-resource scenarios](https://mlsanity.com/paper/2109.06103/) (2021)

## Built on by (most cited first, 4 of 4)

- [Granary: Speech Recognition and Translation Dataset in 25 European Languages](https://mlsanity.com/paper/2505.13404/) (2025)
- [HAINAN: Fast and Accurate Transducer for Hybrid-Autoregressive ASR](https://mlsanity.com/paper/2410.02597/) (2024)
- [Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMs](https://mlsanity.com/paper/2606.29534/) (2026)
- [Whispering Context: Distilling Syntax and Semantics for Long Speech Transcripts](https://mlsanity.com/paper/2508.13376/) (2025)

## Abstract (arXiv)

This paper presents a new method for training sequence-to-sequence models for speech recognition and translation tasks. Instead of the traditional approach of training models on short segments containing only lowercase or partial punctuation and capitalization (PnC) sentences, we propose training on longer utterances that include complete sentences with proper punctuation and capitalization. We achieve this by using the FastConformer architecture which allows training 1 Billion parameter models with sequences up to 60 seconds long with full attention. However, while training with PnC enhances the overall performance, we observed that accuracy plateaus when training on sequences longer than 40 seconds across various evaluation settings. Our proposed method significantly improves punctuation and capitalization accuracy, showing a 25% relative word error rate (WER) improvement on the Earnings-21 and Earnings-22 benchmarks. Additionally, training on longer audio segments increases the overall model accuracy across speech recognition and translation benchmarks. The model weights and training code are open-sourced though NVIDIA NeMo.