# Scaling up masked audio encoder learning for general audio classification

arXiv: https://arxiv.org/abs/2406.06992 · HTML: https://mlsanity.com/paper/2406.06992/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2024.

## Built on by (most cited first, 30 of 44)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning](https://mlsanity.com/paper/2507.02915/) (2025)
- [Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale](https://mlsanity.com/paper/2604.18572/) (2026)
- [GLAP: General contrastive audio-text pretraining across domains and languages](https://mlsanity.com/paper/2506.11350/) (2025)
- [Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification](https://mlsanity.com/paper/2509.24901/) (2025)
- [MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks](https://mlsanity.com/paper/2507.23511/) (2025)
- [MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning](https://mlsanity.com/paper/2508.12709/) (2025)
- [FlexSED: Towards Open-Vocabulary Sound Event Detection](https://mlsanity.com/paper/2509.18606/) (2025)
- [Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation](https://mlsanity.com/paper/2511.16757/) (2025)
- [Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio](https://mlsanity.com/paper/2607.18666/) (2026)
- [FISHER: A Foundation Model for Multi-Modal Industrial Signal Comprehensive Representation](https://mlsanity.com/paper/2507.16696/) (2025)
- [SonicBench: Dissecting the Physical Perception Bottleneck in Large Audio Language Models](https://mlsanity.com/paper/2601.11039/) (2026)
- [MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning](https://mlsanity.com/paper/2606.25225/) (2026)
- [Hashing-Baseline: Rethinking Hashing in the Age of Pretrained Models](https://mlsanity.com/paper/2509.14427/) (2025)
- [SynSonic: Augmenting Sound Event Detection through Text-to-Audio Diffusion ControlNet and Effective Sample Filtering](https://mlsanity.com/paper/2509.18603/) (2025)
- [UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation](https://mlsanity.com/paper/2602.21772/) (2026)
- [Unlocking Strong Supervision: A Data-Centric Study of General-Purpose Audio Pre-Training Methods](https://mlsanity.com/paper/2603.25767/) (2026)
- [USAD 2.0: Scaling Representation Distillation for Universal Audio Understanding](https://mlsanity.com/paper/2606.06444/) (2026)
- [ALM2Vec: Learning Audio Embeddings for Universal Audio Retrieval with Large Audio-Language Models](https://mlsanity.com/paper/2606.30682/) (2026)
- [VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation](https://mlsanity.com/paper/2608.13613/) (2026)
- [Can We Predict Anomaly Detection Performance from Embedding-Space Geometry?](https://mlsanity.com/paper/2609.26460/) (2026)
- [Multimodal Laryngoscopic Video Analysis for Assisted Diagnosis of Vocal Fold Paralysis](https://mlsanity.com/paper/2409.03597/) (2024)
- [Revisiting SSL for sound event detection: complementary fusion and adaptive post-processing](https://mlsanity.com/paper/2505.11889/) (2025)
- [AudioMAE++: learning better masked audio representations with SwiGLU FFNs](https://mlsanity.com/paper/2507.10464/) (2025)
- [ECHO: Frequency-aware Hierarchical Encoding for Variable-length Signals](https://mlsanity.com/paper/2508.14689/) (2025)
- [An overview of neural architectures for self-supervised audio representation learning from masked spectrograms](https://mlsanity.com/paper/2509.18691/) (2025)
- [Unify Variables in Neural Scaling Laws for General Audio Representations via Embedding Effective Rank](https://mlsanity.com/paper/2510.10948/) (2025)
- [Better audio representations are more brain-like: linking model-brain alignment with performance in downstream auditory tasks](https://mlsanity.com/paper/2511.16849/) (2025)
- [LambdaMark: Semantic Audio Watermarking for Robustness and Radioactivity](https://mlsanity.com/paper/2606.21365/) (2026)
- [BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations](https://mlsanity.com/paper/2606.30700/) (2026)

## Abstract (arXiv)

Despite progress in audio classification, a generalization gap remains between speech and other sound domains, such as environmental sounds and music. Models trained for speech tasks often fail to perform well on environmental or musical audio tasks, and vice versa. While self-supervised (SSL) audio representations offer an alternative, there has been limited exploration of scaling both model and dataset sizes for SSL-based general audio classification. We introduce Dasheng, a simple SSL audio encoder, based on the efficient masked autoencoder framework. Trained with 1.2 billion parameters on 272,356 hours of diverse audio, Dasheng obtains significant performance gains on the HEAR benchmark. It outperforms previous works on CREMA-D, LibriCount, Speech Commands, VoxLingua, and competes well in music and environment classification. Dasheng features inherently contain rich speech, music, and environmental information, as shown in nearest-neighbor classification experiments. Code is available https://github.com/richermans/dasheng/.