# MAVL: A Multilingual Audio-Video Lyrics Dataset for Animated Song Translation

arXiv: https://arxiv.org/abs/2505.18614 · HTML: https://mlsanity.com/paper/2505.18614/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 37 references that are in the map.
- First posted on arXiv 24 May 2025.

## Builds on (most cited first, 15 of 15)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [LLaVA-CoT: Let Vision Language Models Reason Step-by-Step](https://mlsanity.com/paper/2411.10440/) (2024)
- [Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model](https://mlsanity.com/paper/2501.07246/) (2025)
- [VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Tool](https://mlsanity.com/paper/2407.05355/) (2024)
- [Multi-modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2312.08762/) (2023)
- [Chain-of-Thought Prompting for Speech Translation](https://mlsanity.com/paper/2409.11538/) (2024)
- [Automatic Song Translation for Tonal Languages](https://mlsanity.com/paper/2203.13420/) (2022)
- [Songs Across Borders: Singable and Controllable Neural Lyric Translation](https://mlsanity.com/paper/2305.16816/) (2023)
- [Large Language Model for Multi-Domain Translation: Benchmarking and Domain CoT Fine-tuning](https://mlsanity.com/paper/2410.02631/) (2024)
- [Translate the Beauty in Songs: Jointly Learning to Align Melody and Translate Lyrics](https://mlsanity.com/paper/2303.15705/) (2023)
- [A Computational Evaluation Framework for Singable Lyric Translation](https://mlsanity.com/paper/2308.13715/) (2023)
- [K-pop Lyric Translation: Dataset, Analysis, and Neural-Modelling](https://mlsanity.com/paper/2309.11093/) (2023)
- [Sing it, Narrate it: Quality Musical Lyrics Translation](https://mlsanity.com/paper/2410.22066/) (2024)

## Abstract (arXiv)

Lyrics translation requires both accurate semantic transfer and preservation of musical rhythm, syllabic structure, and poetic style. In animated musicals, the challenge intensifies due to alignment with visual and auditory cues. We introduce Multilingual Audio-Video Lyrics Benchmark for Animated Song Translation (MAVL), the first multilingual, multimodal benchmark for singable lyrics translation. By integrating text, audio, and video, MAVL enables richer and more expressive translations than text-only approaches. Building on this, we propose Syllable-Constrained Audio-Video LLM with Chain-of-Thought SylAVL-CoT, which leverages audio-video cues and enforces syllabic constraints to produce natural-sounding lyrics. Experimental results demonstrate that SylAVL-CoT significantly outperforms text-based models in singability and contextual accuracy, emphasizing the value of multimodal, multilingual approaches for lyrics translation.