# Adapting pretrained speech model for Mandarin lyrics transcription and alignment

arXiv: https://arxiv.org/abs/2311.12488 · HTML: https://mlsanity.com/paper/2311.12488/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2023.

## Built on by (most cited first, 4 of 4)

- [Lyrics Transcription for Humans: A Readability-Aware Benchmark](https://mlsanity.com/paper/2408.06370/) (2024)
- [Towards Building an End-to-End Multilingual Automatic Lyrics Transcription Model](https://mlsanity.com/paper/2406.17618/) (2024)
- [VocalParse: Towards Unified and Scalable Singing Voice Transcription with Large Audio Language Models](https://mlsanity.com/paper/2605.04613/) (2026)
- [MPEcho: A Melody and Phoneme-Aware Generative Framework for Controllable Cover Song Generation](https://mlsanity.com/paper/2607.26698/) (2026)

## Abstract (arXiv)

The tasks of automatic lyrics transcription and lyrics alignment have witnessed significant performance improvements in the past few years. However, most of the previous works only focus on English in which large-scale datasets are available. In this paper, we address lyrics transcription and alignment of polyphonic Mandarin pop music in a low-resource setting. To deal with the data scarcity issue, we adapt pretrained Whisper model and fine-tune it on a monophonic Mandarin singing dataset. With the use of data augmentation and source separation model, results show that the proposed method achieves a character error rate of less than 18% on a Mandarin polyphonic dataset for lyrics transcription, and a mean absolute error of 0.071 seconds for lyrics alignment. Our results demonstrate the potential of adapting a pretrained speech model for lyrics transcription and alignment in low-resource scenarios.