mlsanityData to 7 Oct 2026

MuDiT & MuSiT: Alignment with Colloquial Expression in Description-to-Song Generation

arXiv 2407.03188 ↗First posted 3 Jul 2024cs.SD cs.AI cs.MMcited by 2

Not on the map. Why?

Abstract (arXiv)

Amid the rising intersection of generative AI and human artistic processes, this study probes the critical yet less-explored terrain of alignment in human-centric automatic song composition. We propose a novel task of Colloquial Description-to-Song Generation, which focuses on aligning the generated content with colloquial human expressions. This task is aimed at bridging the gap between colloquial language understanding and auditory expression within an AI model, with the ultimate goal of creating songs that accurately satisfy human auditory expectations and structurally align with musical norms. Current datasets are limited due to their narrow descriptive scope, semantic gaps and inaccuracies. To overcome data scarcity in this domain, we present the Caichong Music Dataset (CaiMD). CaiMD is manually annotated by both professional musicians and amateurs, offering diverse perspectives and a comprehensive understanding of colloquial descriptions. Unlike existing datasets pre-set with expert annotations or auto-generated ones with inherent biases, CaiMD caters more sufficiently to our purpose of aligning AI-generated music with widespread user-desired results. Moreover, we propose an innovative single-stage framework called MuDiT/MuSiT for enabling effective human-machine alignment in song creation. This framework not only achieves cross-modal comprehension between colloquial language and auditory music perceptions but also ensures generated songs align with user-desired results. MuDiT/MuSiT employs one DiT/SiT model for end-to-end generation of musical components like melody, harmony, rhythm, vocals, and instrumentation. The approach ensures harmonious sonic cohesiveness amongst all generated musical components, facilitating better resonance with human auditory expectations.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 38 of its 47 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

38 of 47 references.

References · 38

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Scalable Diffusion Models with Transformerscited by 5,931
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2023LIMA: Less Is More for Alignmentcited by 1,080
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  9. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  10. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  11. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 707
  12. 2022High Fidelity Neural Audio Compressioncited by 673
26 more
  1. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  2. 2020Jukebox: A Generative Model for Musiccited by 535
  3. 2023Simple and Controllable Music Generationcited by 480
  4. 2023MusicLM: Generating Music From Textcited by 442
  5. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  6. 2023Noise2Music: Text-conditioned Music Generation with Diffusion Modelscited by 173
  7. 2022Hybrid Transformers for Music Source Separationcited by 134
  8. 2021Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphscited by 109
  9. 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
  10. 2021Hybrid Spectrogram and Waveform Source Separationcited by 90
  11. 2021MusicBERT: Symbolic Music Understanding with Large-Scale Pre-Trainingcited by 89
  12. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 85
  13. 2020POP909: A Pop-song Dataset for Music Arrangement Generationcited by 83
  14. 2023Mousai: Text-to-Music Generation with Long-Context Latent Diffusioncited by 77
  15. 2017MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generationcited by 72
  16. 2023MuseCoco: Generating Symbolic Music from Textcited by 50
  17. 2022Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generationcited by 48
  18. 2020SongMASS: Automatic Song Writing with Pre-training and Alignment Constraintcited by 41
  19. 2024On the Scalability of Diffusion-based Text-to-Image Generationcited by 36
  20. 2023ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformercited by 24
  21. 2021TeleMelody: Lyric-to-Melody Generation with a Template-Based Two-Stage Methodcited by 20
  22. 2023ERNIE-Music: Text-to-Waveform Music Generation with Diffusion Modelscited by 15
  23. 2024MuChin: A Chinese Colloquial Description Benchmark for Evaluating Language Models in the Field of Musiccited by 15
  24. 2022SongDriver: Real-time Music Accompaniment Generation without Logical Latency nor Exposure Biascited by 13
  25. 2024Text-to-Song: Towards Controllable Music Generation Incorporating Vocals and Accompanimentcited by 13
  26. 2023MelodyGLM: Multi-task Pre-training for Symbolic Melody Generationcited by 10

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2025A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectivescited by 4
  2. 2024SongGLM: Lyric-to-Melody Generation with 2D Alignment Encoding and Multi-Task Pre-Trainingcited by 1