AnimeScore: A Preference-Based Dataset and Framework for Evaluating Anime-Like Speech Style
Not on the map. Why?
Abstract (arXiv)
Evaluating 'anime-like' voices currently relies on costly subjective judgments, yet no standardized objective metric exists. A key challenge is that anime-likeness, unlike naturalness, lacks a shared absolute scale, making conventional Mean Opinion Score (MOS) protocols unreliable. To address this gap, we propose AnimeScore, a preference-based framework for automatic anime-likeness evaluation via pairwise ranking. We collect 15,000 pairwise judgments from 187 evaluators with free-form descriptions, and acoustic analysis reveals that perceived anime-likeness is driven by controlled resonance shaping, prosodic continuity, and deliberate articulation rather than simple heuristics such as high pitch. We show that handcrafted acoustic features reach a 69.3% AUC ceiling, while SSL-based ranking models achieve up to 90.8% AUC, providing a practical metric that can also serve as a reward signal for preference-based optimization of generative speech models.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 30 references that are in the map. First posted on arXiv 12 Mar 2026.
Builds on
23 of 30 references.
References · 23
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
- 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
- 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
- 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
- 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
- 2025Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Soundcited by 129
- 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
- 2021Generalization Ability of MOS Prediction Networkscited by 69
11 more
- 2022The VoiceMOS Challenge 2022cited by 57
- 2024SpeechAlign: Aligning Speech Generation to Human Preferencescited by 45
- 2025Sidon: Fast and Robust Open-Source Multilingual Speech Restoration for Large-scale Dataset Cleansingcited by 8
- 2023MOSPC: MOS Prediction Based on Pairwise Comparisoncited by 7
- 2023Coco-Nut: Corpus of Japanese Utterance and Voice Characteristics Description for Prompt-based Controlcited by 7
- 2024ANIM-400K: A Large-Scale Dataset for Automated End-To-End Dubbing of Videocited by 6
- 2025From Scores to Preferences: Redefining MOS Benchmarking for Speech Quality Reward Modelingcited by 4
- 2024SAMOS: A Neural MOS Prediction Model Leveraging Semantic Representations and Acoustic Featurescited by 2
- 2025APG-MOS: Auditory Perception Guided-MOS Predictor for Synthetic Speechcited by 2
- 2025Universal Preference-Score-based Pairwise Speech Quality Assessmentcited by 2
- 2025Multi-Sampling-Frequency Naturalness MOS Prediction Using Self-Supervised Learning Model with Sampling-Frequency-Independent Layercited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.