# Mel-RoFormer for Vocal Separation and Vocal Melody Transcription

arXiv: https://arxiv.org/abs/2409.04702 · HTML: https://mlsanity.com/paper/2409.04702/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Sep 2024.

## Built on by (most cited first, 8 of 8)

- [Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation](https://mlsanity.com/paper/2605.08729/) (2026)
- [YingMusic-SVC: Real-World Robust Zero-Shot Singing Voice Conversion with Flow-GRPO and Singing-Specific Inductive Biases](https://mlsanity.com/paper/2512.04793/) (2025)
- [JAVEDIT: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation](https://mlsanity.com/paper/2606.03168/) (2026)
- [Improving Music Source Separation with Diffusion and Consistency Refinement](https://mlsanity.com/paper/2412.06965/) (2024)
- [Diff-VS: Efficient Audio-Aware Diffusion U-Net for Vocals Separation](https://mlsanity.com/paper/2604.01120/) (2026)
- [DTT-BSR+: A Generative-Regression Cascade for Music Source Restoration](https://mlsanity.com/paper/2606.24127/) (2026)
- [Audio-to-Score Transcription using Pre-trained Features, Data Augmentation, and the New SheetSage-A2S Dataset](https://mlsanity.com/paper/2608.06165/) (2026)
- [Vocal Music under Phoneme-Conditional Analysis](https://mlsanity.com/paper/2608.30823/) (2026)

## Abstract (arXiv)

Developing a versatile deep neural network to model music audio is crucial in MIR. This task is challenging due to the intricate spectral variations inherent in music signals, which convey melody, harmonics, and timbres of diverse instruments. In this paper, we introduce Mel-RoFormer, a spectrogram-based model featuring two key designs: a novel Mel-band Projection module at the front-end to enhance the model's capability to capture informative features across multiple frequency bands, and interleaved RoPE Transformers to explicitly model the frequency and time dimensions as two separate sequences. We apply Mel-RoFormer to tackle two essential MIR tasks: vocal separation and vocal melody transcription, aimed at isolating singing voices from audio mixtures and transcribing their lead melodies, respectively. Despite their shared focus on singing signals, these tasks possess distinct optimization objectives. Instead of training a unified model, we adopt a two-step approach. Initially, we train a vocal separation model, which subsequently serves as a foundation model for fine-tuning for vocal melody transcription. Through extensive experiments conducted on benchmark datasets, we showcase that our models achieve state-of-the-art performance in both vocal separation and melody transcription tasks, underscoring the efficacy and versatility of Mel-RoFormer in modeling complex music audio signals.