# DALI: a large Dataset of synchronized Audio, LyrIcs and notes, automatically created using teacher-student machine learning paradigm

arXiv: https://arxiv.org/abs/1906.10606 · HTML: https://mlsanity.com/paper/1906.10606/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2019.

## Built on by (most cited first, 19 of 19)

- [LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT](https://mlsanity.com/paper/2306.17103/) (2023)
- [Contrastive Learning-Based Audio to Lyrics Alignment for Multiple Languages](https://mlsanity.com/paper/2306.07744/) (2023)
- [MSTRE-Net: Multistreaming Acoustic Modeling for Automatic Lyrics Transcription](https://mlsanity.com/paper/2108.02625/) (2021)
- [Genre-conditioned Acoustic Models for Automatic Lyrics Transcription of Polyphonic Music](https://mlsanity.com/paper/2204.03307/) (2022)
- [A Phoneme-Informed Neural Network Model for Note-Level Singing Transcription](https://mlsanity.com/paper/2304.05917/) (2023)
- [Lyrics Transcription for Humans: A Readability-Aware Benchmark](https://mlsanity.com/paper/2408.06370/) (2024)
- [And what if two musical versions don't share melody, harmony, rhythm, or lyrics ?](https://mlsanity.com/paper/2210.01256/) (2022)
- [Automatic Time Signature Determination for New Scores Using Lyrics for Latent Rhythmic Structure](https://mlsanity.com/paper/2311.15480/) (2023)
- [The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibration](https://mlsanity.com/paper/2502.01458/) (2025)
- [Self-refining of Pseudo Labels for Music Source Separation with Noisy Labeled Data](https://mlsanity.com/paper/2307.12576/) (2023)
- [More than words: Advancements and challenges in speech recognition for singing](https://mlsanity.com/paper/2403.09298/) (2024)
- [Song Form-aware Full-Song Text-to-Lyrics Generation with Multi-Level Granularity Syllable Count Control](https://mlsanity.com/paper/2411.13100/) (2024)
- [LyriCAR: A Difficulty-Aware Curriculum Reinforcement Learning Framework For Controllable Lyric Translation](https://mlsanity.com/paper/2510.19967/) (2025)
- [Towards Context-Aware Neural Performance-Score Synchronisation](https://mlsanity.com/paper/2206.00454/) (2022)
- [A Real-Time Lyrics Alignment System Using Chroma And Phonetic Features For Classical Vocal Performance](https://mlsanity.com/paper/2401.09200/) (2024)
- [VietLyrics: A Large-Scale Dataset and Models for Vietnamese Automatic Lyrics Transcription](https://mlsanity.com/paper/2510.22295/) (2025)
- [Scalable Music Cover Retrieval Using Lyrics-Aligned Audio Embeddings](https://mlsanity.com/paper/2601.11262/) (2026)
- [Enhancing Automatic Chord Recognition via Pseudo-Labeling and Knowledge Distillation](https://mlsanity.com/paper/2602.19778/) (2026)
- [Automatic Lyric Transcription for Greek Songs: Scaling and Task Composition Effects in Whisper Adaptation](https://mlsanity.com/paper/2609.11302/) (2026)

## Abstract (arXiv)

The goal of this paper is twofold. First, we introduce DALI, a large and rich multimodal dataset containing 5358 audio tracks with their time-aligned vocal melody notes and lyrics at four levels of granularity. The second goal is to explain our methodology where dataset creation and learning models interact using a teacher-student machine learning paradigm that benefits each other. We start with a set of manual annotations of draft time-aligned lyrics and notes made by non-expert users of Karaoke games. This set comes without audio. Therefore, we need to find the corresponding audio and adapt the annotations to it. To that end, we retrieve audio candidates from the Web. Each candidate is then turned into a singing-voice probability over time using a teacher, a deep convolutional neural network singing-voice detection system (SVD), trained on cleaned data. Comparing the time-aligned lyrics and the singing-voice probability, we detect matches and update the time-alignment lyrics accordingly. From this, we obtain new audio sets. They are then used to train new SVD students used to perform again the above comparison. The process could be repeated iteratively. We show that this allows to progressively improve the performances of our SVD and get better audio-matching and alignment.