mlsanityData to 14 Oct 2026

Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecs

arXiv 2511.16639 ↗First posted 20 Nov 2025eess.AS cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Recent advancements in neural audio codecs have not only enabled superior audio compression but also enhanced speech synthesis techniques. Researchers are now exploring their potential as universal acoustic feature extractors for a broader range of speech processing tasks. Building on this trend, we introduce Codec2Vec, the first speech representation learning framework that relies exclusively on discrete audio codec units. This approach offers several advantages, including improved data storage and transmission efficiency, faster training, and enhanced data privacy. We explore masked prediction with various training target derivation strategies to thoroughly understand the effectiveness of this framework. Evaluated on the SUPERB benchmark, Codec2Vec achieves competitive performance compared to continuous-input models while reducing storage requirements by up to 16.5x and training time by 2.3x, showcasing its scalability and efficiency.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 33 of its 35 references that are in the map. First posted on arXiv 20 Nov 2025.

Builds on

33 of 35 references.

References · 33

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  6. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
  7. 2022High Fidelity Neural Audio Compressioncited by 687
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  9. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  10. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  11. 2024The Faiss librarycited by 376
  12. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
21 more
  1. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  2. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  3. 2024VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wildcited by 120
  4. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  5. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  6. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  7. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  8. 2023SpeechX: Neural Codec Language Model as a Versatile Speech Transformercited by 64
  9. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  10. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 40
  11. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 37
  12. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  13. 2024Codec-SUPERB: An In-Depth Analysis of Sound Codec Modelscited by 26
  14. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  15. 2022MelHuBERT: A simplified HuBERT on Mel spectrogramscited by 16
  16. 2023Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognitioncited by 11
  17. 2023EnCodecMAE: Leveraging neural codecs for universal audio representation learningcited by 9
  18. 2024Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representationscited by 8
  19. 2022CoBERT: Self-Supervised Speech Representation Learning Through Code Representation Learningcited by 7
  20. 2024Sustainable self-supervised learning for speech representationscited by 4
  21. 2025Probing the Robustness Properties of Neural Speech Codecscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Quantizer-Aware Hierarchical Neural Codec Modeling for Speech Deepfake Detectioncited by 0
  2. 2026End-to-End Training for Discrete Token LLM based TTS Systemcited by 0
  3. 2026Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokenscited by 0