# Scalable Music Cover Retrieval Using Lyrics-Aligned Audio Embeddings

arXiv: https://arxiv.org/abs/2601.11262 · HTML: https://mlsanity.com/paper/2601.11262/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 43 of its 64 references that are in the map.
- First posted on arXiv 16 Jan 2026.

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Multilingual E5 Text Embeddings: A Technical Report](https://mlsanity.com/paper/2402.05672/) (2024)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://mlsanity.com/paper/2407.19669/) (2024)
- [MuLan: A Joint Embedding of Music Audio and Natural Language](https://mlsanity.com/paper/2208.12415/) (2022)
- [Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers](https://mlsanity.com/paper/2307.03183/) (2023)
- [Careless Whisper: Speech-to-Text Hallucination Harms](https://mlsanity.com/paper/2402.08021/) (2024)
- [musicnn: Pre-trained convolutional neural networks for music audio tagging](https://mlsanity.com/paper/1909.06654/) (2019)
- [Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio](https://mlsanity.com/paper/2501.11378/) (2025)
- [Augmenting Convolutional networks with attention-based aggregation](https://mlsanity.com/paper/2112.13692/) (2021)
- [Hallucinations in Neural Automatic Speech Recognition: Identifying Errors and Hallucinatory Models](https://mlsanity.com/paper/2401.01572/) (2024)
- [DALI: a large Dataset of synchronized Audio, LyrIcs and notes, automatically created using teacher-student machine learning paradigm](https://mlsanity.com/paper/1906.10606/) (2019)
- [T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining](https://mlsanity.com/paper/2404.17806/) (2024)
- [Audio-based Musical Version Identification: Elements and Challenges](https://mlsanity.com/paper/2109.02472/) (2021)
- [Beyond Transcription: Mechanistic Interpretability in ASR](https://mlsanity.com/paper/2508.15882/) (2025)
- [Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down](https://mlsanity.com/paper/2505.12969/) (2025)
- [Contrastive Learning-Based Audio to Lyrics Alignment for Multiple Languages](https://mlsanity.com/paper/2306.07744/) (2023)
- [GLAP: General contrastive audio-text pretraining across domains and languages](https://mlsanity.com/paper/2506.11350/) (2025)
- [ByteCover: Cover Song Identification via Multi-Loss Training](https://mlsanity.com/paper/2010.14022/) (2020)
- [Learning a Representation for Cover Song Identification Using Convolutional Neural Network](https://mlsanity.com/paper/1911.00334/) (2019)
- [Supervised contrastive learning from weakly-labeled audio segments for musical version matching](https://mlsanity.com/paper/2502.16936/) (2025)
- [Accurate and Scalable Version Identification Using Musically-Motivated Embeddings](https://mlsanity.com/paper/1910.12551/) (2019)

## Abstract (arXiv)

Music Cover Retrieval, also known as Version Identification, aims to recognize distinct renditions of the same underlying musical work, a task central to catalog management, copyright enforcement, and music retrieval. State-of-the-art approaches have largely focused on harmonic and melodic features, employing increasingly complex audio pipelines designed to be invariant to musical attributes that often vary widely across covers. While effective, these methods demand substantial training time and computational resources. By contrast, lyrics constitute a strong invariant across covers, though their use has been limited by the difficulty of extracting them accurately and efficiently from polyphonic audio. Early methods relied on simple frameworks that limited downstream performance, while more recent systems deliver stronger results but require large models integrated within complex multimodal architectures. We introduce LIVI (Lyrics-Informed Version Identification), an approach that seeks to balance retrieval accuracy with computational efficiency. First, LIVI leverages supervision from state-of-the-art transcription and text embedding models during training to achieve retrieval accuracy on par with--or superior to--harmonic-based systems. Second, LIVI remains lightweight and efficient by removing the transcription step at inference, challenging the dominance of complexity-heavy pipelines.