On the de-duplication of the Lakh MIDI dataset
Not on the map. Why?
Abstract (arXiv)
A large-scale dataset is essential for training a well-generalized deep-learning model. Most such datasets are collected via scraping from various internet sources, inevitably introducing duplicated data. In the symbolic music domain, these duplicates often come from multiple user arrangements and metadata changes after simple editing. However, despite critical issues such as unreliable training evaluation from data leakage during random splitting, dataset duplication has not been extensively addressed in the MIR community. This study investigates the dataset duplication issues regarding Lakh MIDI Dataset (LMD), one of the largest publicly available sources in the symbolic music domain. To find and evaluate the best retrieval method for duplicated data, we employed the Clean MIDI subset of the LMD as a benchmark test set, in which different versions of the same songs are grouped together. We first evaluated rule-based approaches and previous symbolic music retrieval models for de-duplication and also investigated with a contrastive learning-based BERT model with various augmentations to find duplicate files. As a result, we propose three different versions of the filtered list of LMD, which filters out at least 38,134 samples in the most conservative settings among 178,561 files.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 20 Sep 2025.
Builds on
32 of 41 references.
References · 32
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
- 2021Deduplicating Training Data Makes Language Models Bettercited by 714
- 2017MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompanimentcited by 151
- 2021MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Trainingcited by 89
- 2019Cutting Music Source Separation Some Slakh: A Dataset to Study the Impact of Training Data Quality and Quantitycited by 86
- 2020PopMAG: Pop Music Accompaniment Generationcited by 68
- 2019LakhNES: Improving multi-instrumental music generation with cross-domain pre-trainingcited by 54
- 2020MMM : Exploring Conditional Multi-Track Music Generation with the Transformercited by 49
- 2022Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generationcited by 49
20 more
- 2023On the De-duplication of LAION-2Bcited by 47
- 2023Anticipatory Music Transformercited by 36
- 2023miditok: A Python package for MIDI file tokenizationcited by 27
- 2020PiRhDy: Learning Pitch-, Rhythm-, and Dynamics-aware Embeddings for Symbolic Musiccited by 24
- 2025CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languagescited by 21
- 2024MidiCaps: A large-scale MIDI dataset with text captionscited by 19
- 2024Text2midi: Generating Symbolic Music from Captionscited by 19
- 2022Symbolic music generation conditioned on continuous-valued emotionscited by 13
- 2024Exploring Musical Roots: Applying Audio Embeddings to Empower Influence Attribution for a Generative Music Modelcited by 13
- 2024PianoBART: Symbolic Piano Music Generation and Understanding with Large-Scale Pre-Trainingcited by 12
- 2024CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Modelscited by 12
- 2013The GTZAN dataset: Its contents, its faults, their effects on evaluation, and its future usecited by 11
- 2024Towards Assessing Data Replication in Music Generation with Music Similarity Metrics on Raw Audiocited by 10
- 2019MIDI-Sandwich2: RNN-based Hierarchical Multi-modal Fusion Generation VAE networks for multi-track symbolic music generationcited by 8
- 2022YM2413-MDB: A Multi-Instrumental FM Video Game Music Dataset with Emotion Annotationscited by 7
- 2025The GigaMIDI Dataset with Features for Expressive Music Performance Detectioncited by 7
- 2023Impact of time and note duration tokenizations on deep learning symbolic music modelingcited by 6
- 2024Nested Music Transformer: Sequentially Decoding Compound Tokens in Symbolic Music and Audio Generationcited by 5
- 2022Instrument Separation of Symbolic Music by Explicitly Guided Diffusion Modelcited by 2
- 2023Systematic Analysis of Music Representations from BERTcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.