mlsanityData to 14 Oct 2026

How Much Context Does My Attention-Based ASR System Need?

arXiv 2310.15672 ↗First posted 24 Oct 2023cs.CL cs.SD eess.AScited by 3

Not on the map. Why?

Abstract (arXiv)

For the task of speech recognition, the use of more than 30 seconds of acoustic context during training is uncommon and under-investigated in literature. In this work, we conduct an empirical study on the effect of scaling the sequence length used to train/evaluate (dense-attention-based) acoustic models on speech recognition performance. For these experiments, a dataset of roughly 100,000 pseudo-labelled Spotify podcasts is used, with context lengths of 5 seconds to 1 hour being explored. Zero-shot evaluations are presented on the long-format datasets: Earnings-22, Tedlium and Rev16. Results demonstrate a benefit from training with up to 21.8 minutes of acoustic context, showing up to a 14.5% relative improvement from a baseline trained with 10 seconds of context. We find that the model's width/depth, positional encoding scheme and number of attention heads impact its ability to use longer contexts.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 38 of its 44 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

38 of 44 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016Layer Normalizationcited by 5,669
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  10. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  12. 2020GLU Variants Improve Transformercited by 1,804
26 more
  1. 2019Root Mean Square Layer Normalizationcited by 1,480
  2. 2022In-context Learning and Induction Headscited by 964
  3. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  4. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  5. 2020Query-Key Normalization for Transformerscited by 280
  6. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  7. 2021CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attentioncited by 139
  8. 2017Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Modelscited by 128
  9. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 124
  10. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  11. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  12. 2020Rethinking Evaluation in ASR: Are Our Models Robust Enough?cited by 73
  13. 2020Talking-Heads Attentioncited by 62
  14. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  15. 2022Earnings-22: A Practical Benchmark for Accents in the Wildcited by 50
  16. 2019A comparison of end-to-end models for long-form speech recognitioncited by 42
  17. 2021The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Modelscited by 36
  18. 2021Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimizationcited by 28
  19. 2021Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformerscited by 25
  20. 2022ESB: A Benchmark For Multi-Domain End-to-End Speech Recognitioncited by 18
  21. 2021Transformer Language Models with LSTM-based Cross-utterance Information Representationcited by 15
  22. 2023Investigating End-to-End ASR Architectures for Long Form Audio Transcriptioncited by 11
  23. 2021Input Length Matters: Improving RNN-T and MWER Training for Long-form Telephony Speech Recognitioncited by 10
  24. 2021Cross-utterance Reranking Models with BERT and Graph Convolutional Networks for Conversational Speech Recognitioncited by 7
  25. 2021Conformer-based End-to-end Speech Recognition With Rotary Position Embeddingcited by 3
  26. 2023Leveraging Cross-Utterance Context For ASR Decodingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2025Effective Context in Neural Speech Modelscited by 5
  2. 2026Segment Length Matters: A Study of Segment Lengths on Audio Fingerprinting Performancecited by 1
  3. 2024Self-Train Before You Transcribecited by 0