# A Streamlined Encoder/Decoder Architecture for Melody Extraction

arXiv: https://arxiv.org/abs/1810.12947 · HTML: https://mlsanity.com/paper/1810.12947/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2018.

## Built on by (most cited first, 6 of 6)

- [SpecTNT: a Time-Frequency Transformer for Music Audio](https://mlsanity.com/paper/2110.09127/) (2021)
- [YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentation](https://mlsanity.com/paper/2407.04822/) (2024)
- [TONet: Tone-Octave Network for Singing Melody Extraction from Polyphonic Music](https://mlsanity.com/paper/2202.00951/) (2022)
- [Towards Improving Harmonic Sensitivity and Prediction Stability for Singing Melody Extraction](https://mlsanity.com/paper/2308.02723/) (2023)
- [Deep-Learning Architectures for Multi-Pitch Estimation: Towards Reliable Evaluation](https://mlsanity.com/paper/2202.09198/) (2022)
- [A Mamba-based Network for Semi-supervised Singing Melody Extraction Using Confidence Binary Regularization](https://mlsanity.com/paper/2505.08681/) (2025)

## Abstract (arXiv)

Melody extraction in polyphonic musical audio is important for music signal processing. In this paper, we propose a novel streamlined encoder/decoder network that is designed for the task. We make two technical contributions. First, drawing inspiration from a state-of-the-art model for semantic pixel-wise segmentation, we pass through the pooling indices between pooling and un-pooling layers to localize the melody in frequency. We can achieve result close to the state-of-the-art with much fewer convolutional layers and simpler convolution modules. Second, we propose a way to use the bottleneck layer of the network to estimate the existence of a melody line for each time frame, and make it possible to use a simple argmax function instead of ad-hoc thresholding to get the final estimation of the melody line. Our experiments on both vocal melody extraction and general melody extraction validate the effectiveness of the proposed model.