Linear Complexity Self-Supervised Learning for Music Understanding with Random Quantizer
Not on the map. Why?
Abstract (arXiv)
In recent years, foundation models have become very popular due to their exceptional performance, mainly in natural language (NLP) tasks where they were first introduced. These models usually consist of hundreds of millions, or even billions, of parameters, making them resource-intensive during training and in production systems, leading to increased costs. This paper focuses on the reduction of a foundation's model size when applied to music information retrieval (MIR) tasks. Our research combines the Branchformer architecture with SummaryMixing, which were first applied in speech recognition, along with a random quantization process. To facilitate reproducibility, we conduct pre-training on publicly available datasets, complemented by a proprietary dataset comparable in scale to other private datasets reported in the literature. We ensure robust evaluation by using a framework consisting of a variety of downstream MIR tasks. Our results show that our architecture achieves competitive performance when compared with other state-of-the-art models that use multi-head self-attention, while reducing the model size from 8.5% up to 12.3%.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 49 references that are in the map. First posted on arXiv 14 Jan 2026.
Builds on
28 of 49 references.
References · 28
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2017Neural Discrete Representation Learningcited by 4,650
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
- 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
- 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
16 more
- 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
- 2016FMA: A Dataset For Music Analysiscited by 205
- 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
- 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
- 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
- 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
- 2021Contrastive Learning of Musical Representationscited by 81
- 2021Codified audio language modeling learns useful representations for music information retrievalcited by 61
- 2019musicnn: Pre-trained convolutional neural networks for music audio taggingcited by 49
- 2023A Foundation Model for Music Informaticscited by 45
- 2022Supervised and Unsupervised Learning of Audio Representations for Music Understandingcited by 34
- 2017End-to-end learning for music audio tagging at scalecited by 25
- 2015Deep Learning and Music Adversariescited by 20
- 2017End-to-End Musical Key Estimation Using a Convolutional Neural Networkcited by 12
- 2022Deformable CNN and Imbalance-Aware Feature Learning for Singing Technique Classificationcited by 8
- 2025OMAR-RQ: Open Music Audio Representation Model Trained with Multi-Feature Masked Token Predictioncited by 5
Built on by
Not cited yet.