mlsanityData to 14 Oct 2026

SingMOS-Pro: An Comprehensive Benchmark for Singing Quality Assessment

arXiv 2510.01812 ↗First posted 2 Oct 2025cs.SD cs.AI eess.AScited by 7

Not on the map. Why?

Abstract (arXiv)

Singing voice generation progresses rapidly, yet evaluating singing quality remains a critical challenge. Human subjective assessment, typically in the form of listening tests, is costly and time consuming, while existing objective metrics capture only limited perceptual aspects. In this work, we introduce SingMOS-Pro, a dataset for automatic singing quality assessment. Building on our preview version SingMOS, which provides only overall ratings, SingMOS-Pro extends the annotations of the additional data to include lyrics, melody, and overall quality, offering broader coverage and greater diversity. The dataset contains 7,981 singing clips generated by 41 models across 12 datasets, spanning from early systems to recent state-of-the-art approaches. Each clip is rated by at least five experienced annotators to ensure reliability and consistency. Furthermore, we investigate strategies for effectively utilizing MOS data annotated under heterogeneous standards and benchmark several widely used evaluation methods from related tasks on SingMOS-Pro, establishing strong baselines and practical references for future research. The dataset is publicly available at https://huggingface.co/datasets/TangRain/SingMOS-Pro.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 44 of its 48 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

44 of 48 references.

References · 44

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  3. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  4. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  7. 2022High Fidelity Neural Audio Compressioncited by 687
  8. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  9. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 431
  10. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 390
  11. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  12. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 340
32 more
  1. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 241
  2. 2021DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanismcited by 154
  3. 2025YuE: Scaling Open Foundation Models for Long-Form Music Generationcited by 71
  4. 2024ChatMusician: Understanding and Generating Music Intrinsically with LLMcited by 68
  5. 2025ACE-Step: A Step Towards Music Generation Foundation Modelcited by 56
  6. 2022Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesiscited by 53
  7. 2025DiffRhythm: Blazingly Fast and Embarrassingly Simple End-to-End Full-Length Song Generation with Latent Diffusioncited by 46
  8. 2021VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesiscited by 38
  9. 2020XiaoiceSing: A High-Quality and Integrated Singing Voice Synthesis Systemcited by 33
  10. 2024GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Taskscited by 30
  11. 2024ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speechcited by 26
  12. 2023StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesiscited by 21
  13. 2024TCSinger: Zero-Shot Singing Voice Synthesis with Style Transfer and Multi-Level Style Controlcited by 20
  14. 2025TechSinger: Technique Controllable Multilingual Singing Voice Synthesis via Flow Matchingcited by 19
  15. 2023Mel-Band RoFormer for Music Source Separationcited by 17
  16. 2024The VoiceMOS Challenge 2024: Beyond Speech Quality Predictioncited by 17
  17. 2024CtrSVDD: A Benchmark Dataset and Baseline Analysis for Controlled Singing Voice Deepfake Detectioncited by 16
  18. 2020JVS-MuSiC: Japanese multispeaker singing-voice corpuscited by 14
  19. 2024MOS-Bench: Benchmarking Generalization Abilities of Subjective Speech Quality Assessment Modelscited by 12
  20. 2022Muskits: an End-to-End Music Processing Toolkit for Singing Voice Synthesiscited by 10
  21. 2024SingMOS: An extensive Open-Source Singing Voice Dataset for MOS Predictioncited by 9
  22. 2021VocBench: A Neural Vocoder Benchmark for Speech Synthesiscited by 8
  23. 2020Sequence-to-sequence Singing Voice Synthesis with Perceptual Entropy Losscited by 6
  24. 2024TokSing: Singing Voice Synthesis based on Discrete Tokenscited by 6
  25. 2024Singing Voice Data Scaling-up: An Introduction to ACE-Opencpop and ACE-KiSingcited by 4
  26. 2022NNSVS: A Neural Network-Based Singing Voice Synthesis Toolkitcited by 3
  27. 2024Pitch-and-Spectrum-Aware Singing Quality Assessment with Bias Correction and Model Fusioncited by 3
  28. 2023A Comparative Study of Voice Conversion Models with Large-Scale Speech and Singing Data: The T13 Systems for the Singing Voice Conversion Challenge 2023cited by 2
  29. 2024VISinger2+: End-to-End Singing Voice Synthesis Augmented by Self-Supervised Learning Representationcited by 2
  30. 2024SingOMD: Singing Oriented Multi-resolution Discrete Representation Construction from Speech Modelscited by 2
  31. 2024AlignNet: Learning dataset score alignment functions to enable better training of speech quality estimatorscited by 2
  32. 2024Muskits-ESPnet: A Comprehensive Toolkit for Singing Voice Synthesis in New Paradigmcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026TuneJury: An Open Metric for Improving Music Generation Preference Alignmentcited by 4
  2. 2026SoulX-Singer: Towards High-Quality Zero-Shot Singing Voice Synthesiscited by 3
  3. 2026VidAudio-Bench: Benchmarking V2A and VT2A Generation across Four Audio Categoriescited by 1
  4. 2026VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singingcited by 1
  5. 2026Controllable Singing Style Conversion with Boundary-Aware Information Bottleneckcited by 0
  6. 2026VocalParse: Towards Unified and Scalable Singing Voice Transcription with Large Audio Language Modelscited by 0
  7. 2026Listening Like a Judge: A Music-Aware Framework for Automatic Singing Performance Evaluationcited by 0