# MAJL: A Model-Agnostic Joint Learning Framework for Music Source Separation and Pitch Estimation

arXiv: https://arxiv.org/abs/2501.03689 · HTML: https://mlsanity.com/paper/2501.03689/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 70 references that are in the map.
- First posted on arXiv 7 Jan 2025.

## Builds on (most cited first, 29 of 29)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation](https://mlsanity.com/paper/1806.03185/) (2018)
- [CREPE: A Convolutional Representation for Pitch Estimation](https://mlsanity.com/paper/1802.06182/) (2018)
- [Music Source Separation in the Waveform Domain](https://mlsanity.com/paper/1911.13254/) (2019)
- [Onsets and Frames: Dual-Objective Piano Transcription](https://mlsanity.com/paper/1710.11153/) (2017)
- [Hybrid Spectrogram and Waveform Source Separation](https://mlsanity.com/paper/2111.03600/) (2021)
- [MT3: Multi-Task Multitrack Music Transcription](https://mlsanity.com/paper/2111.03017/) (2021)
- [Music Source Separation with Band-split RNN](https://mlsanity.com/paper/2209.15174/) (2022)
- [Decoupling Magnitude and Phase Estimation with Deep ResUNet for Music Source Separation](https://mlsanity.com/paper/2109.05418/) (2021)
- [RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Music](https://mlsanity.com/paper/2306.15412/) (2023)
- [SPICE: Self-supervised Pitch Estimation](https://mlsanity.com/paper/1910.11664/) (2019)
- [Zero-shot Audio Source Separation through Query-based Learning from Weakly-labeled Data](https://mlsanity.com/paper/2112.07891/) (2021)
- [Multimodal Pretraining, Adaptation, and Generation for Recommendation: A Survey](https://mlsanity.com/paper/2404.00621/) (2024)
- [KUIELab-MDX-Net: A Two-Stream Neural Network for Music Demixing](https://mlsanity.com/paper/2111.12203/) (2021)
- [Improved Speech Enhancement with the Wave-U-Net](https://mlsanity.com/paper/1811.11307/) (2018)
- [PMG : Personalized Multimodal Generation with Large Language Models](https://mlsanity.com/paper/2404.08677/) (2024)
- [A Unified Model for Zero-shot Music Source Separation, Transcription and Synthesis](https://mlsanity.com/paper/2108.03456/) (2021)
- [Deep Polyphonic ADSR Piano Note Transcription](https://mlsanity.com/paper/1906.09165/) (2019)
- [Adversarial Learning for Improved Onsets and Frames Music Transcription](https://mlsanity.com/paper/1906.08512/) (2019)
- [Transcription Is All You Need: Learning to Separate Musical Mixtures with Score as Supervision](https://mlsanity.com/paper/2010.11904/) (2020)
- [CWS-PResUNet: Music Source Separation with Channel-wise Subband Phase-aware ResUNet](https://mlsanity.com/paper/2112.04685/) (2021)
- [DEEPF0: End-To-End Fundamental Frequency Estimation for Music and Speech Signals](https://mlsanity.com/paper/2102.06306/) (2021)
- [ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding](https://mlsanity.com/paper/2311.11375/) (2023)
- [An Overview of Lead and Accompaniment Separation in Music](https://mlsanity.com/paper/1804.08300/) (2018)
- [HarmoF0: Logarithmic Scale Dilated Convolution For Pitch Estimation](https://mlsanity.com/paper/2205.01019/) (2022)
- [JEPOO: Highly Accurate Joint Estimation of Pitch, Onset and Offset for Music Information Retrieval](https://mlsanity.com/paper/2306.01304/) (2023)
- [A Tutorial on Independent Component Analysis](https://mlsanity.com/paper/1404.2986/) (2014)
- [Improving singing voice separation with the Wave-U-Net using Minimum Hyperspherical Energy](https://mlsanity.com/paper/1910.10071/) (2019)
- [SpaIn-Net: Spatially-Informed Stereophonic Music Source Separation](https://mlsanity.com/paper/2202.07523/) (2022)

## Abstract (arXiv)

Music source separation and pitch estimation are two vital tasks in music information retrieval. Typically, the input of pitch estimation is obtained from the output of music source separation. Therefore, existing methods have tried to perform these two tasks simultaneously, so as to leverage the mutually beneficial relationship between both tasks. However, these methods still face two critical challenges that limit the improvement of both tasks: the lack of labeled data and joint learning optimization. To address these challenges, we propose a Model-Agnostic Joint Learning (MAJL) framework for both tasks. MAJL is a generic framework and can use variant models for each task. It includes a two-stage training method and a dynamic weighting method named Dynamic Weights on Hard Samples (DWHS), which addresses the lack of labeled data and joint learning optimization, respectively. Experimental results on public music datasets show that MAJL outperforms state-of-the-art methods on both tasks, with significant improvements of 0.92 in Signal-to-Distortion Ratio (SDR) for music source separation and 2.71% in Raw Pitch Accuracy (RPA) for pitch estimation. Furthermore, comprehensive studies not only validate the effectiveness of each component of MAJL, but also indicate the great generality of MAJL in adapting to different model architectures.