mlsanityData to 14 Oct 2026

Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning

arXiv 2507.02915 ↗First posted 25 Jun 2025cs.SD cs.AI cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Building on the Joint-Embedding Predictive Architecture (JEPA) paradigm, a recent self-supervised learning framework that predicts latent representations of masked regions in high-level feature spaces, we propose Audio-JEPA (Audio Joint-Embedding Predictive Architecture), tailored specifically for audio data. Audio-JEPA uses a simple Vision Transformer backbone to predict latent representations of masked spectrogram patches rather than reconstructing raw audio. We pre-train on unlabeled AudioSet clips (10s, 32kHz) with random patch masking on mel-spectrograms. We evaluate on the X-ARES suite covering speech, music, and environmental sound tasks. Although our implementation is a straightforward translation of the original model to audio, the results still show comparable performance to wav2vec 2.0 and data2vec while using less than one-fifth of their training data and with no hyper-parameter tuning. All code and pretrained checkpoints are available on GitHub.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 25 of its 41 references that are in the map. First posted on arXiv 25 Jun 2025.

Builds on

25 of 41 references.

References · 25

  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  6. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  7. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  8. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  9. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  10. 2019Clotho: An Audio Captioning Datasetcited by 358
  11. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  12. 2022Masked Autoencoders that Listencited by 273
13 more
  1. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 225
  2. 2016FMA: A Dataset For Music Analysiscited by 207
  3. 2022MetaFormer Baselines for Visioncited by 132
  4. 2022Vocalsound: A Dataset for Improving Human Vocal Sounds Recognitioncited by 62
  5. 2023A-JEPA: Joint-Embedding Predictive Architecture Can Listencited by 53
  6. 2024Scaling up masked audio encoder learning for general audio classificationcited by 44
  7. 2018General-purpose Tagging of Freesound Audio with AudioSet Labels: Task Description, Dataset, and Baselinecited by 30
  8. 2023CED: Consistent ensemble distillation for audio taggingcited by 29
  9. 2024Masked Modeling Duo: Towards a Universal Audio Pre-training Frameworkcited by 23
  10. 2023ConvFormer: Plug-and-Play CNN-Style Transformers for Improving Medical Image Segmentationcited by 9
  11. 2024Stem-JEPA: A Joint-Embedding Predictive Architecture for Musical Stem Compatibility Estimationcited by 8
  12. 2025TI-JEPA: An Innovative Energy-based Joint Embedding Strategy for Text-Image Multimodal Systemscited by 8
  13. 2025GeoJEPA: Towards Eliminating Augmentation- and Sampling Bias in Multimodal Geospatial Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 16.

17 papers.

Sort:

By area:

  1. 2025Context-level Language Modeling by Learning Predictive Context Embeddingscited by 6
  2. 2026Self-Distillation of Hidden Layers for Self-Supervised Representation Learningcited by 3
  3. 2026Pantagruel: Unified Self-Supervised Encoders for French Text and Speechcited by 2
  4. 2026Music-JEPA: Learning a World Model of Sound from Actioncited by 2
  5. 2026The JEPA Paradox in Language: The Geometry of Linguistic Alternativescited by 2
  6. 2026Joint-Embedding Predictive Architecture for Sensor-based Activity Recognitioncited by 1
  7. 2026SJEPA: Learning Elegant Latent Dynamics with Hybrid Symbolic-Neural Predictorscited by 1
  8. 2026World-as-Graph: Relational World Modeling Through Latent Space Graphscited by 1
  9. 2026Echo: A Joint-Embedding Predictive Architecture for Speaker Diarization and Speech Recognition in a Shared Latent Spacecited by 0
  10. 2026ArtNet: A JEPA-Like Articulatory Predictive Framework for Robust Zero-Shot Phoneme Recognitioncited by 0
  11. 2026BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representationscited by 0
  12. 2026The JEPA Predictor: A Transferable Operator for Occluded Feature Completioncited by 0
5 more
  1. 2026Bar-JEPA: Extracting Values from Bar Chart with Joint-Embedding Predictive Architecturecited by 0
  2. 2026Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learnerscited by 0
  3. 2026Human-JEPA: A Human-Centric Vision Model that Perceives and Anticipatescited by 0
  4. 2026Predictive audio representations for early detection and tracking of hidden dynamic objectscited by 0
  5. 2026GLaS-JEPA: Gaussian-Regularized Speech SSL without Engineered Prediction Targetscited by 0