# SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations

arXiv: https://arxiv.org/abs/2510.25955 · HTML: https://mlsanity.com/paper/2510.25955/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2025.

## Built on by (most cited first, 10 of 10)

- [MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning](https://mlsanity.com/paper/2606.25225/) (2026)
- [Alethia: A Foundational Encoder for Voice Deepfakes](https://mlsanity.com/paper/2605.00251/) (2026)
- [USAD 2.0: Scaling Representation Distillation for Universal Audio Understanding](https://mlsanity.com/paper/2606.06444/) (2026)
- [NVMOS: Non-Verbal Vocalization Quality Assessment in Speech](https://mlsanity.com/paper/2606.15888/) (2026)
- [SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents](https://mlsanity.com/paper/2609.34247/) (2026)
- [STEP: Scientific Time-Series Encoder Pretraining via Cross-Domain Distillation](https://mlsanity.com/paper/2603.18688/) (2026)
- [Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners](https://mlsanity.com/paper/2608.19863/) (2026)
- [Bearings: Self-Supervised Soundfield Embeddings from First-Order Ambisonics](https://mlsanity.com/paper/2609.23152/) (2026)
- [VoiceNet: Fine-Grained Voice Understanding Beyond Emotion at Scale](https://mlsanity.com/paper/2609.32016/) (2026)
- [YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality](https://mlsanity.com/paper/2609.33757/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) has significantly advanced acoustic representation learning. However, most existing models are optimised for either speech or audio event understanding, resulting in a persistent gap between these two domains. We address this gap with SPEAR (SPEech and Audio Representations), a self-supervised framework that distils complementary knowledge from a speech-focused SSL teacher and a general-audio SSL teacher into a single unified model. SPEAR applies multi-codebook vector quantisation to continuous teacher representations to produce fine-grained discrete tokens that capture both semantic and acoustic information. To effectively integrate these heterogeneous representations, SPEAR jointly predicts them given a masked input with an asymmetric pre-training loss. We further improve robustness in complex sound scenes through a novel token mixing mechanism. Extensive experiments demonstrate that SPEAR consistently outperforms existing unified speech and audio models. SPEAR establishes a new state-of-the-art on the SUPERB benchmark, surpassing WavLM Large on 12 of 15 tasks, while achieving competitive performance on the HEAR benchmark. These results position SPEAR as a versatile foundation for general-purpose speech and audio representation learning. The code and pre-trained models will be released.