mlsanityData to 14 Oct 2026

On-device Streaming Discrete Speech Units

arXiv 2506.01845 ↗First posted 2 Jun 2025eess.AS cs.LG cs.SDcited by 4

Not on the map. Why?

Abstract (arXiv)

Discrete speech units (DSUs) are derived from clustering the features of self-supervised speech models (S3Ms). DSUs offer significant advantages for on-device streaming speech applications due to their rich phonetic information, high transmission efficiency, and seamless integration with large language models. However, conventional DSU-based approaches are impractical as they require full-length speech input and computationally expensive S3Ms. In this work, we reduce both the attention window and the model size while preserving the effectiveness of DSUs. Our results demonstrate that we can reduce floating-point operations (FLOPs) by 50% with only a relative increase of 6.5% in character error rate (CER) on the ML-SUPERB 1h dataset. These findings highlight the potential of DSUs for real-time speech processing in resource-constrained environments.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 31 of its 40 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

31 of 40 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Neural Discrete Representation Learningcited by 4,712
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  7. 2022High Fidelity Neural Audio Compressioncited by 687
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  9. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  10. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  11. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  12. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
19 more
  1. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  2. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  3. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  4. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  5. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  6. 2020Streaming automatic speech recognition with the transformer modelcited by 73
  7. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 71
  8. 2024Self-Supervised Speech Representations are More Phonetic than Semanticcited by 71
  9. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  10. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  11. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 31
  12. 2023Exploration of Efficient End-to-End ASR using Discretized Input from Self-Supervised Learningcited by 28
  13. 2024ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speechcited by 26
  14. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 25
  15. 2025Leveraging Allophony in Self-Supervised Speech Models for Atypical Pronunciation Assessmentcited by 15
  16. 2024The Interspeech 2024 Challenge on Speech Processing Using Discrete Unitscited by 12
  17. 2023Understanding Probe Behaviors through Variational Bounds of Mutual Informationcited by 11
  18. 2023Distilling HuBERT with LSTMs via Decoupled Knowledge Distillationcited by 4
  19. 2025Discrete Speech Unit Extraction via Independent Component Analysiscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillationcited by 9
  2. 2026Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspacescited by 8
  3. 2025WhisperRT -- Turning Whisper into a Causal Streaming Modelcited by 0
  4. 2026Frontend Token Enhancement for Token-Based Speech Recognitioncited by 0