# SongEval: A Benchmark Dataset for Song Aesthetics Evaluation

arXiv: https://arxiv.org/abs/2505.10793 · HTML: https://mlsanity.com/paper/2505.10793/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2025.

## Built on by (most cited first, 26 of 26)

- [ACE-Step: A Step Towards Music Generation Foundation Model](https://mlsanity.com/paper/2506.00045/) (2025)
- [DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching](https://mlsanity.com/paper/2510.22950/) (2025)
- [Muse: Towards Reproducible Long-Form Song Generation with Fine-Grained Style Control](https://mlsanity.com/paper/2601.03973/) (2026)
- [MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloning](https://mlsanity.com/paper/2601.01568/) (2026)
- [SongBench: A Fine-Grained Multi-Aspect Benchmark for Song Quality Assessment](https://mlsanity.com/paper/2604.25937/) (2026)
- [MuQ-Eval: An Open-Source Per-Sample Quality Metric for AI Music Generation Evaluation](https://mlsanity.com/paper/2603.22677/) (2026)
- [LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Training](https://mlsanity.com/paper/2606.30642/) (2026)
- [InstructAudio: Unified speech and music generation with natural language instruction](https://mlsanity.com/paper/2511.18487/) (2025)
- [CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction](https://mlsanity.com/paper/2603.00610/) (2026)
- [TuneJury: An Open Metric for Improving Music Generation Preference Alignment](https://mlsanity.com/paper/2606.17006/) (2026)
- [Pushing the Frontier of Full-Song Generation: Hierarchical Autoregressive Planning Meets Flow-Matching Rendering](https://mlsanity.com/paper/2607.20253/) (2026)
- [Zero-Effort Image-to-Music Generation: An Interpretable RAG-based VLM Approach](https://mlsanity.com/paper/2509.22378/) (2025)
- [Aligning Generative Music AI with Human Preferences: Methods and Challenges](https://mlsanity.com/paper/2511.15038/) (2025)
- [Hear: Hierarchically Enhanced Aesthetic Representations For Multidimensional Music Evaluation](https://mlsanity.com/paper/2511.18869/) (2025)
- [Tutti: Expressive Multi-Singer Synthesis via Structure-Level Timbre Control and Vocal Texture Modeling](https://mlsanity.com/paper/2602.08233/) (2026)
- [SyncTrack: Rhythmic Stability and Synchronization in Multi-Track Music Generation](https://mlsanity.com/paper/2603.01101/) (2026)
- [LaDA-Band: Language Diffusion Models for Vocal-to-Accompaniment Generation](https://mlsanity.com/paper/2604.11052/) (2026)
- [APEX: Large-scale Multi-task Aesthetic-Informed Popularity Prediction for AI-Generated Music](https://mlsanity.com/paper/2605.03395/) (2026)
- [SegTune: Structured and Fine-Grained Control for Song Generation](https://mlsanity.com/paper/2606.02638/) (2026)
- [SketchSong: Hierarchical Song Generation with Sketch Planning and Fine-Grained Multi-Track Modeling](https://mlsanity.com/paper/2606.03169/) (2026)
- [Towards Unified Song Generation and Singing Voice Conversion with Accompaniment Co-Generation](https://mlsanity.com/paper/2606.07015/) (2026)
- [Improving Text-to-Music Generation with Human Preference Rewards](https://mlsanity.com/paper/2606.21670/) (2026)
- [WanSong v1.0 Technical Report](https://mlsanity.com/paper/2607.14749/) (2026)
- [MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques](https://mlsanity.com/paper/2608.11755/) (2026)
- [YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality](https://mlsanity.com/paper/2609.33757/) (2026)
- [What Do Audio-Language Judges of Generated Music Listen For? Four Probes with Controls from Expert Ratings](https://mlsanity.com/paper/2610.13247/) (2026)

## Abstract (arXiv)

Aesthetics serve as an implicit and important criterion in song generation tasks that reflect human perception beyond objective metrics. However, evaluating the aesthetics of generated songs remains a fundamental challenge, as the appreciation of music is highly subjective. Existing evaluation metrics, such as embedding-based distances, are limited in reflecting the subjective and perceptual aspects that define musical appeal. To address this issue, we introduce SongEval, the first open-source, large-scale benchmark dataset for evaluating the aesthetics of full-length songs. SongEval includes over 2,399 songs in full length, summing up to more than 140 hours, with aesthetic ratings from 16 professional annotators with musical backgrounds. Each song is evaluated across five key dimensions: overall coherence, memorability, naturalness of vocal breathing and phrasing, clarity of song structure, and overall musicality. The dataset covers both English and Chinese songs, spanning nine mainstream genres. Moreover, to assess the effectiveness of song aesthetic evaluation, we conduct experiments using SongEval to predict aesthetic scores and demonstrate better performance than existing objective evaluation metrics in predicting human-perceived musical quality.