mlsanityData to 13 Oct 2026

Music Foundation Model as Generic Booster for Music Downstream Tasks

arXiv 2411.01135 ↗First posted 2 Nov 2024cs.SD cs.IR cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

We demonstrate the efficacy of using intermediate representations from a single foundation model to enhance various music downstream tasks. We introduce SoniDo, a music foundation model (MFM) designed to extract hierarchical features from target music samples. By leveraging hierarchical intermediate features, SoniDo constrains the information granularity, leading to improved performance across various downstream tasks including both understanding and generative tasks. We specifically evaluated this approach on representative tasks such as music tagging, music transcription, music source separation, and music mixing. Our results reveal that the features extracted from foundation models provide valuable enhancements in training downstream task models. This highlights the capability of using features extracted from music foundation models as a booster for downstream tasks. Our approach not only benefits existing task-specific models but also supports music downstream tasks constrained by data scarcity. This paves the way for more effective and accessible music processing solutions.

Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Builds on 62 of its 98 references that are in the map. First posted on arXiv 2 Nov 2024.

Builds on

62 of 98 references.

References · 62

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  4. 2020Language Models are Few-Shot Learnerscited by 32,320
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
  7. 2017Squeeze-and-Excitation Networkscited by 4,772
  8. 2017Neural Discrete Representation Learningcited by 4,680
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,313
  10. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,306
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,538
  12. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,381
38 more
  1. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,359
  2. 2021SoundStream: An End-to-End Neural Audio Codeccited by 762
  3. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 721
  4. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 713
  5. 2022Autoregressive Image Generation using Residual Quantizationcited by 709
  6. 2022High Fidelity Neural Audio Compressioncited by 680
  7. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 596
  8. 2020Jukebox: A Generative Model for Musiccited by 537
  9. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 223
  10. 2021Efficient Training of Audio Transformers with Patchoutcited by 223
  11. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 222
  12. 2022Hybrid Transformers for Music Source Separationcited by 134
  13. 2017Onsets and Frames: Dual-Objective Piano Transcriptioncited by 94
  14. 2021Contrastive Learning of Musical Representationscited by 81
  15. 2021MT3: Multi-Task Multitrack Music Transcriptioncited by 70
  16. 2022SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantizationcited by 68
  17. 2016Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applicationscited by 61
  18. 2020High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Timescited by 61
  19. 2021Codified audio language modeling learns useful representations for music information retrievalcited by 61
  20. 2016Learning Features of Music from Scratchcited by 58
  21. 2022Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representationcited by 52
  22. 2021Towards Learning Universal Audio Representationscited by 51
  23. 2023Efficient Neural Music Generationcited by 50
  24. 2019musicnn: Pre-trained convolutional neural networks for music audio taggingcited by 49
  25. 2021Music Demixing Challenge 2021cited by 38
  26. 2023MARBLE: Music Audio Representation Benchmark for Universal Evaluationcited by 38
  27. 2023Music Source Separation with Band-Split RoPE Transformercited by 37
  28. 2022Investigating self-supervised learning for speech enhancement and separationcited by 35
  29. 2024Foundation Models for Music: A Surveycited by 35
  30. 2022Supervised and Unsupervised Learning of Audio Representations for Music Understandingcited by 34
  31. 2019Conditioned-U-Net: Introducing a Control Mechanism in the U-Net for Multiple Source Separationscited by 27
  32. 2020Automatic multitrack mixing with a differentiable mixing console of neural audio effectscited by 27
  33. 2023HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayescited by 27
  34. 2022Boosting Self-Supervised Embeddings for Speech Enhancementcited by 24
  35. 2022Melody transcription via generative pre-trainingcited by 23
  36. 2023Automatic Piano Transcription with Hierarchical Frequency-Time Transformercited by 22
  37. 2021ReconVAT: A Semi-Supervised Automatic Music Transcription Framework for Low-Resource Real-World Datacited by 20
  38. 2023The Sound Demixing Challenge 2023 unicodex2013 Music Demixing Trackcited by 20

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNetcited by 5
  2. 2025Noise-to-Notes: Diffusion-based Generation and Refinement for Automatic Drum Transcriptioncited by 1
  3. 2025Do Foundational Audio Encoders Understand Music Structure?cited by 1
  4. 2025Investigating an Overfitting and Degeneration Phenomenon in Self-Supervised Multi-Pitch Estimationcited by 0
  5. 2025Improving BERT for Symbolic Music Understanding Using Token Denoising and Pianoroll Predictioncited by 0
  6. 2026What Makes a Good Layer? Assessing the Layer-Wise Intrinsic Properties of Music Foundation Modelscited by 0