mlsanityData to 14 Oct 2026

Effective Context in Neural Speech Models

arXiv 2505.22487 ↗First posted 28 May 2025cs.SD cs.CL eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

Modern neural speech models benefit from having longer context, and many approaches have been proposed to increase the maximum context a model can use. However, few have attempted to measure how much context these models actually use, i.e., the effective context. Here, we propose two approaches to measuring the effective context, and use them to analyze different speech Transformers. For supervised models, we find that the effective context correlates well with the nature of the task, with fundamental frequency tracking, phone classification, and word classification requiring increasing amounts of effective context. For self-supervised models, we find that effective context increases mainly in the early layers, and remains relatively short -- similar to the supervised phone model. Given that these models do not use a long context during prediction, we show that HuBERT can be run in streaming mode without modification to the architecture and without further fine-tuning.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 29 of its 37 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

29 of 37 references.

References · 29

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  11. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
  12. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
17 more
  1. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  2. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  3. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  4. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  5. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 152
  6. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  7. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 91
  8. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
  9. 2023What Do Self-Supervised Speech Models Know About Words?cited by 64
  10. 2021What Context Features Can Transformer Language Models Use?cited by 50
  11. 2022Probing phoneme, language and speaker information in unsupervised speech representationscited by 33
  12. 2020On the Usefulness of Self-Attention for Automatic Speech Recognition with Transformerscited by 24
  13. 2020Improving Streaming Automatic Speech Recognition With Non-Streaming Model Distillation On Unsupervised Datacited by 16
  14. 2024Orthogonality and isotropy of speaker and phonetic information in self-supervised speech representationscited by 14
  15. 2024XLSR-Transducer: Streaming ASR for Self-Supervised Pretrained Modelscited by 6
  16. 2018On Training Recurrent Networks with Truncated Backpropagation Through Time in Speech Recognitioncited by 5
  17. 2023How Much Context Does My Attention-Based ASR System Need?cited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillationcited by 9
  2. 2026Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspacescited by 8
  3. 2026Reconstruct! Don't Encode: Self-Supervised Representation Reconstruction Loss for High-Intelligibility and Low-Latency Streaming Neural Audio Codeccited by 2
  4. 2026Perceptual compensation for tonal context in self-supervised speech modelscited by 0
  5. 2026Quieter Than the Room: Representation Drift and Task Robustness in Speech Encoderscited by 0