# Unify Variables in Neural Scaling Laws for General Audio Representations via Embedding Effective Rank

arXiv: https://arxiv.org/abs/2510.10948 · HTML: https://mlsanity.com/paper/2510.10948/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 38 of its 53 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Builds on (most cited first, 30 of 38)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank](https://mlsanity.com/paper/2210.02885/) (2022)
- [Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training](https://mlsanity.com/paper/2104.01027/) (2021)
- [Self-Supervised Representation Learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2110.09327/) (2021)
- [Scaling Laws for Generative Mixed-Modal Language Models](https://mlsanity.com/paper/2301.03728/) (2023)
- [UniAudio: An Audio Foundation Model Toward Universal Audio Generation](https://mlsanity.com/paper/2310.00704/) (2023)
- [Scaling Laws of Synthetic Images for Model Training ... for Now](https://mlsanity.com/paper/2312.04567/) (2023)
- [HEAR: Holistic Evaluation of Audio Representations](https://mlsanity.com/paper/2203.03022/) (2022)
- [Audio-Visual Segmentation with Semantics](https://mlsanity.com/paper/2301.13190/) (2023)
- [MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization](https://mlsanity.com/paper/2501.01108/) (2025)
- [ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning](https://mlsanity.com/paper/2101.10803/) (2021)
- [Scaling Laws for Downstream Task Performance of Large Language Models](https://mlsanity.com/paper/2402.04177/) (2024)
- [Scaling up masked audio encoder learning for general audio classification](https://mlsanity.com/paper/2406.06992/) (2024)
- [SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Model](https://mlsanity.com/paper/2405.11831/) (2024)

## Abstract (arXiv)

Scaling laws have profoundly shaped our understanding of model performance in computer vision and natural language processing, yet their application to general audio representation learning remains underexplored. A key challenge lies in the multifactorial nature of general audio representation-representation quality is jointly influenced by variables such as audio length, embedding dimensionality, model depth, model architecture, data volume, etc., many of which are difficult to isolate or express analytically. In this work, we present a systematic study of scaling laws for general audio representations by utilizing embedding effective rank (RankMe) as a unifying metric that encapsulates the impact of diverse variables on representation quality. RankMe enables a label-free, information-theoretic quantification of audio embeddings, allowing us to examine scaling behaviors across a wide hyper-parameter space, including model size, training data volume, computational budget, architectural configurations, etc. Our empirical findings reveal a consistent power-law relationship between RankMe and representation quality, suggesting that embedding effective rank serves as a reliable proxy for assessing and predicting model performance in audio representation learning. This work not only validates the applicability of classical scaling principles to the general audio domain but also offers a theoretically grounded and empirically robust framework for guiding future model scaling strategies in audio foundation models.