mlsanityData to 7 Oct 2026

Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representations

arXiv 2407.03495 ↗First posted 3 Jul 2024eess.AS cs.CL cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Discrete speech representations have garnered recent attention for their efficacy in training transformer-based models for various speech-related tasks such as automatic speech recognition (ASR), translation, speaker verification, and joint speech-text foundational models. In this work, we present a comprehensive analysis on building ASR systems with discrete codes. We investigate different methods for codec training such as quantization schemes and time-domain vs spectral feature encodings. We further explore ASR training techniques aimed at enhancing performance, training efficiency, and noise robustness. Drawing upon our findings, we introduce a codec ASR pipeline that outperforms Encodec at similar bit-rate. Remarkably, it also surpasses the state-of-the-art results achieved by strong self-supervised models on the 143 languages ML-SUPERB benchmark despite being smaller in size and pretrained on significantly less data.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 33 of its 38 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

33 of 38 references.

References · 33

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  8. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  9. 2022High Fidelity Neural Audio Compressioncited by 673
  10. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  11. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  12. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
21 more
  1. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  2. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  3. 2023End-to-End Speech Recognition: A Surveycited by 127
  4. 2021A Review of Speaker Diarization: Recent Advances with Deep Learningcited by 122
  5. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  6. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 120
  7. 2017TasNet: time-domain audio separation network for real-time, single-channel speech separationcited by 115
  8. 2023SoundStorm: Efficient Parallel Audio Generationcited by 111
  9. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
  10. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  11. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  12. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 70
  13. 2021LEAF: A Learnable Frontend for Audio Classificationcited by 69
  14. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  15. 2023AudioDec: An Open-source Streaming High-fidelity Neural Audio Codeccited by 47
  16. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  17. 2023Exploration of Efficient End-to-End ASR using Discretized Input from Self-Supervised Learningcited by 28
  18. 2023RepCodec: A Speech Representation Codec for Speech Tokenizationcited by 27
  19. 2023Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognitioncited by 11
  20. 2024Spectral Codecs: Improving Non-Autoregressive Speech Synthesis with Spectrogram-Based Audio Codecscited by 7
  21. 2024The X-LANCE Technical Report for Interspeech 2024 Speech Processing Using Discrete Speech Unit Challengecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 2.

8 papers.

  1. 2025Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecscited by 3
  2. 2025Analysing the Language of Neural Audio Codecscited by 2
  3. 2025Finite Scalar Quantization Enables Redundant and Transmission-Robust Neural Audio Compression at Low Bit-ratescited by 2
  4. 2025UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Modelscited by 1
  5. 2025Explainable Disentanglement on Discrete Speech Representations for Noise-Robust ASRcited by 1
  6. 2026Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditionscited by 1
  7. 2025Robust Residual Finite Scalar Quantization for Neural Compressioncited by 0
  8. 2026MELD: Mel-Spectrogram-Based Speech Language Modeling with Discrete Latent Variablescited by 0