# Automated Conversion of Music Videos into Lyric Videos

arXiv: https://arxiv.org/abs/2308.14922 · HTML: https://mlsanity.com/paper/2308.14922/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 52 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 7 of 7)

- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [TOOD: Task-aligned One-stage Object Detection](https://mlsanity.com/paper/2108.07755/) (2021)
- [A Local-to-Global Approach to Multi-modal Movie Scene Segmentation](https://mlsanity.com/paper/2004.02678/) (2020)
- [Robust Visual Tracking by Segmentation](https://mlsanity.com/paper/2203.11191/) (2022)
- [Unifying Identification and Context Learning for Person Recognition](https://mlsanity.com/paper/1806.03084/) (2018)
- [A Comprehensive Survey on Video Saliency Detection with Auditory Information: the Audio-visual Consistency Perceptual is the Key!](https://mlsanity.com/paper/2206.13390/) (2022)
- [Speaker-following Video Subtitles](https://mlsanity.com/paper/1407.5145/) (2014)

## Built on by (most cited first, 4 of 4)

- [Generative AI for Film Creation: A Survey of Recent Advances](https://mlsanity.com/paper/2504.08296/) (2025)
- [ExpressEdit: Video Editing with Natural Language and Sketching](https://mlsanity.com/paper/2403.17693/) (2024)
- [ScriptViz: A Visualization Tool to Aid Scriptwriting based on a Large Movie Database](https://mlsanity.com/paper/2410.03224/) (2024)
- [ComVi: Context-Aware Optimized Comment Display in Video Playback](https://mlsanity.com/paper/2603.26173/) (2026)

## Abstract (arXiv)

Musicians and fans often produce lyric videos, a form of music videos that showcase the song's lyrics, for their favorite songs. However, making such videos can be challenging and time-consuming as the lyrics need to be added in synchrony and visual harmony with the video. Informed by prior work and close examination of existing lyric videos, we propose a set of design guidelines to help creators make such videos. Our guidelines ensure the readability of the lyric text while maintaining a unified focus of attention. We instantiate these guidelines in a fully automated pipeline that converts an input music video into a lyric video. We demonstrate the robustness of our pipeline by generating lyric videos from a diverse range of input sources. A user study shows that lyric videos generated by our pipeline are effective in maintaining text readability and unifying the focus of attention.