# WavJEPA: Semantic learning unlocks robust audio foundation models for raw waveforms

arXiv: https://arxiv.org/abs/2509.23238 · HTML: https://mlsanity.com/paper/2509.23238/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2025.

## Built on by (most cited first, 7 of 7)

- [Self-Distillation of Hidden Layers for Self-Supervised Representation Learning](https://mlsanity.com/paper/2603.15553/) (2026)
- [Pantagruel: Unified Self-Supervised Encoders for French Text and Speech](https://mlsanity.com/paper/2601.05911/) (2026)
- [Spectro-Temporal Interference Confounds Phase Encoding in Spatial Audio Foundation Models](https://mlsanity.com/paper/2606.14820/) (2026)
- [SpeechDx: A Multi-Task Benchmark for Clinical Speech AI](https://mlsanity.com/paper/2606.17339/) (2026)
- [OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL](https://mlsanity.com/paper/2606.30356/) (2026)
- [Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners](https://mlsanity.com/paper/2608.19863/) (2026)
- [GLaS-JEPA: Gaussian-Regularized Speech SSL without Engineered Prediction Targets](https://mlsanity.com/paper/2609.37798/) (2026)

## Abstract (arXiv)

Learning audio representations from raw waveforms overcomes key limitations of spectrogram-based audio representation learning, such as the long latency of spectrogram computation and the loss of phase information. Yet, while self-supervised speech representation learning from raw waveforms has been remarkably successful, these approaches have not achieved similar feats for general-purpose audio representation learning from waveforms. Here, we propose WavJEPA, a waveform-based version of the Joint-Embedding Predictive Architecture. WavJEPA leverages high-level semantic representation learning to tackle the shortcomings of representation learning at the speech unit or token level. We show that this approach substantially outperforms state-of-the-art time-domain audio foundation models across a wide variety of downstream benchmark tasks, while requiring considerably fewer computational resources. Additionally, to overcome the performance drop that time-domain models typically exhibit in noisy and reverberant real-world acoustic environments, we present WavJEPA-Nat. WavJEPA-Nat is a multi-channel extension of the WavJEPA architecture trained on simulated naturalistic scenes. We find that WavJEPA-Nat is highly robust to reverberation and noise. These results highlight the feasibility and computational efficiency of general-purpose audio representation learning from raw waveforms, showcasing the potential for low-latency, robust time-domain audio foundation models for real-world applications.