# Multimodal Lyrics-Rhythm Matching

arXiv: https://arxiv.org/abs/2301.02732 · HTML: https://mlsanity.com/paper/2301.02732/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 26 references that are in the map.
- First posted on arXiv 6 Jan 2023.

## Builds on (most cited first, 4 of 4)

- [Phone-to-audio alignment without text: A Semi-supervised Approach](https://mlsanity.com/paper/2110.03876/) (2021)
- [End-to-end Lyrics Alignment for Polyphonic Music Using an Audio-to-Character Recognition Model](https://mlsanity.com/paper/1902.06797/) (2019)
- [Improving Lyrics Alignment through Joint Pitch Detection](https://mlsanity.com/paper/2202.01646/) (2022)
- [Structure-Aware Audio-to-Score Alignment using Progressively Dilated Convolutional Neural Networks](https://mlsanity.com/paper/2102.00382/) (2021)

## Built on by (most cited first, 3 of 3)

- [Automatic Time Signature Determination for New Scores Using Lyrics for Latent Rhythmic Structure](https://mlsanity.com/paper/2311.15480/) (2023)
- [Relationships between Keywords and Strong Beats in Lyrical Music](https://mlsanity.com/paper/2412.04202/) (2024)
- [MusicAIR: A Multimodal AI Music Generation Framework Powered by an Algorithm-Driven Core](https://mlsanity.com/paper/2511.17323/) (2025)

## Abstract (arXiv)

Despite the recent increase in research on artificial intelligence for music, prominent correlations between key components of lyrics and rhythm such as keywords, stressed syllables, and strong beats are not frequently studied. This is likely due to challenges such as audio misalignment, inaccuracies in syllabic identification, and most importantly, the need for cross-disciplinary knowledge. To address this lack of research, we propose a novel multimodal lyrics-rhythm matching approach in this paper that specifically matches key components of lyrics and music with each other without any language limitations. We use audio instead of sheet music with readily available metadata, which creates more challenges yet increases the application flexibility of our method. Furthermore, our approach creatively generates several patterns involving various multimodalities, including music strong beats, lyrical syllables, auditory changes in a singer's pronunciation, and especially lyrical keywords, which are utilized for matching key lyrical elements with key rhythmic elements. This advantageous approach not only provides a unique way to study auditory lyrics-rhythm correlations including efficient rhythm-based audio alignment algorithms, but also bridges computational linguistics with music as well as music cognition. Our experimental results reveal an 0.81 probability of matching on average, and around 30% of the songs have a probability of 0.9 or higher of keywords landing on strong beats, including 12% of the songs with a perfect landing. Also, the similarity metrics are used to evaluate the correlation between lyrics and rhythm. It shows that nearly 50% of the songs have 0.70 similarity or higher. In conclusion, our approach contributes significantly to the lyrics-rhythm relationship by computationally unveiling insightful correlations.