# Leveraging Whisper Embeddings for Audio-based Lyrics Matching

arXiv: https://arxiv.org/abs/2510.08176 · HTML: https://mlsanity.com/paper/2510.08176/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Builds on (most cited first, 12 of 12)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Fine-tuning CNN Image Retrieval with No Human Annotation](https://mlsanity.com/paper/1711.02512/) (2017)
- [LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT](https://mlsanity.com/paper/2306.17103/) (2023)
- [Audio-based Musical Version Identification: Elements and Challenges](https://mlsanity.com/paper/2109.02472/) (2021)
- [Learning a Representation for Cover Song Identification Using Convolutional Neural Network](https://mlsanity.com/paper/1911.00334/) (2019)
- [Supervised contrastive learning from weakly-labeled audio segments for musical version matching](https://mlsanity.com/paper/2502.16936/) (2025)
- [ByteCover3: Accurate Cover Song Identification on Short Queries](https://mlsanity.com/paper/2303.11692/) (2023)
- [Large-Scale Cover Song Detection in Digital Music Libraries Using Metadata, Lyrics and Audio Features](https://mlsanity.com/paper/1808.10351/) (2018)
- [Discogs-VI: A Musical Version Identification Dataset Based on Public Editorial Metadata](https://mlsanity.com/paper/2410.17400/) (2024)

## Built on by (most cited first, 2 of 2)

- [Scalable Music Cover Retrieval Using Lyrics-Aligned Audio Embeddings](https://mlsanity.com/paper/2601.11262/) (2026)
- [On the Human and Computer Alignment of Attribute-Based Music Matches](https://mlsanity.com/paper/2609.00987/) (2026)

## Abstract (arXiv)

Audio-based lyrics matching can be an appealing alternative to other content-based retrieval approaches, but existing methods often suffer from limited reproducibility and inconsistent baselines. In this work, we introduce WEALY, a fully reproducible pipeline that leverages Whisper decoder embeddings for lyrics matching tasks. WEALY establishes robust and transparent baselines, while also exploring multimodal extensions that integrate textual and acoustic features. Through extensive experiments on standard datasets, we demonstrate that WEALY achieves a performance comparable to state-of-the-art methods that lack reproducibility. In addition, we provide ablation studies and analyses on language robustness, loss functions, and embedding strategies. This work contributes a reliable benchmark for future research, and underscores the potential of speech technologies for music information retrieval tasks.