mlsanityData to 14 Oct 2026

EnCodecMAE: Leveraging neural codecs for universal audio representation learning

arXiv 2309.07391 ↗First posted 14 Sep 2023cs.SD cs.LG eess.AScited by 9

Not on the map. Why?

Abstract (arXiv)

The goal of universal audio representation learning is to obtain foundational models that can be used for a variety of downstream tasks involving speech, music and environmental sounds. To approach this problem, methods inspired by works on self-supervised learning for NLP, like BERT, or computer vision, like masked autoencoders (MAE), are often adapted to the audio domain. In this work, we propose masking representations of the audio signal, and training a MAE to reconstruct the masked segments. The reconstruction is done by predicting the discrete units generated by EnCodec, a neural audio codec, from the unmasked inputs. We evaluate this approach, which we call EnCodecMAE, on a wide range of tasks involving speech, music and environmental sounds. Our best model outperforms various state-of-the-art audio representation models in terms of global performance. Additionally, we evaluate the resulting representations in the challenging task of automatic speech recognition (ASR), obtaining decent results and paving the way for a universal audio representation.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 34 of its 42 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

34 of 42 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  7. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  8. 2022High Fidelity Neural Audio Compressioncited by 687
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  10. 2021AST: Audio Spectrogram Transformercited by 643
  11. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  12. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
22 more
  1. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  2. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  3. 2022Masked Autoencoders that Listencited by 273
  4. 2021Efficient Training of Audio Transformers with Patchoutcited by 226
  5. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 225
  6. 2016FMA: A Dataset For Music Analysiscited by 207
  7. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 169
  8. 2018CREPE: A Convolutional Representation for Pitch Estimationcited by 110
  9. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  10. 2020Towards Learning a Universal Non-Semantic Representation of Speechcited by 84
  11. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  12. 2022HEAR: Holistic Evaluation of Audio Representationscited by 78
  13. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  14. 2022Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representationcited by 52
  15. 2021Towards Learning Universal Audio Representationscited by 51
  16. 2022Masked Spectrogram Prediction For Self-Supervised Audio Pre-Trainingcited by 51
  17. 2017Unsupervised Learning of Semantic Audio Representationscited by 44
  18. 2023ResiDual: Transformer with Dual Residual Connectionscited by 31
  19. 2021SERAB: A multi-lingual benchmark for speech emotion recognitioncited by 22
  20. 2022The Efficacy of Self-Supervised Speech Models for Audio Representationscited by 6
  21. 2022Learning Music Representations with wav2vec 2.0cited by 4
  22. 2022SLICER: Learning universal audio representations using low-resource self-supervised pre-trainingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 2.

9 papers.

  1. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  2. 2024Foundation Models for Music: A Surveycited by 37
  3. 2024Music2Latent: Consistency Autoencoders for Latent Audio Compressioncited by 33
  4. 2025Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classificationcited by 7
  5. 2024Leveraging Pre-Trained Autoencoders for Interpretable Prototype Learning of Music Audiocited by 5
  6. 2025Codec2Vec: Self-Supervised Speech Representation Learning Using Neural Speech Codecscited by 3
  7. 2025Better audio representations are more brain-like: linking model-brain alignment with performance in downstream auditory taskscited by 0
  8. 2026SqueezeComposer: Temporal Speed-up is A Simple Trick for Long-form Music Composingcited by 0
  9. 2026Multimodal Music Recommendation System using LLMscited by 0