mlsanityData to 7 Oct 2026

Translate the Beauty in Songs: Jointly Learning to Align Melody and Translate Lyrics

arXiv 2303.15705 ↗First posted 28 Mar 2023cs.CL cs.SD eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

Song translation requires both translation of lyrics and alignment of music notes so that the resulting verse can be sung to the accompanying melody, which is a challenging problem that has attracted some interests in different aspects of the translation process. In this paper, we propose Lyrics-Melody Translation with Adaptive Grouping (LTAG), a holistic solution to automatic song translation by jointly modeling lyrics translation and lyrics-melody alignment. It is a novel encoder-decoder framework that can simultaneously translate the source lyrics and determine the number of aligned notes at each decoding step through an adaptive note grouping module. To address data scarcity, we commissioned a small amount of training data annotated specifically for this task and used large amounts of augmented data through back-translation. Experiments conducted on an English-Chinese song translation data set show the effectiveness of our model in both automatic and human evaluation.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 20 of its 41 references that are in the map. First posted on arXiv 28 Mar 2023.

Builds on

20 of 41 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  5. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  6. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  7. 2017Lexically Constrained Decoding for Sequence Generation Using Grid Beam Searchcited by 234
  8. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  9. 2018Achieving Human Parity on Automatic Chinese to English News Translationcited by 119
  10. 2020SongMASS: Automatic Song Writing with Pre-training and Alignment Constraintcited by 41
  11. 2019Controlling the Output Length of Neural Machine Translationcited by 36
  12. 2020DeepSinger: Singing Voice Synthesis with Data Mined From the Webcited by 32
8 more
  1. 2019Conditional LSTM-GAN for Melody Generation from Lyricscited by 31
  2. 2021Controllable Generation from Pre-trained Language Models via Inverse Promptingcited by 29
  3. 2020SongNet: Rigid Formats Controlled Text Generationcited by 25
  4. 2021DeepRapper: Neural Rap Generation with Rhyme and Rhythm Modelingcited by 24
  5. 2021TeleMelody: Lyric-to-Melody Generation with a Template-Based Two-Stage Methodcited by 20
  6. 2020Melody-Conditioned Lyrics Generation with SeqGANscited by 14
  7. 2022Automatic Song Translation for Tonal Languagescited by 12
  8. 2022Learning the Beauty in Songs: Neural Singing Voice Beautifiercited by 10

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2; 2025: 2.

6 papers.

  1. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  2. 2023A Computational Evaluation Framework for Singable Lyric Translationcited by 4
  3. 2023K-pop Lyric Translation: Dataset, Analysis, and Neural-Modellingcited by 4
  4. 2024Sing it, Narrate it: Quality Musical Lyrics Translationcited by 2
  5. 2025LyriCAR: A Difficulty-Aware Curriculum Reinforcement Learning Framework For Controllable Lyric Translationcited by 1
  6. 2025MAVL: A Multilingual Audio-Video Lyrics Dataset for Animated Song Translationcited by 0