mlsanityData to 7 Oct 2026

SpeechLMScore: Evaluating speech generation using speech language model

arXiv 2212.04559 ↗First posted 8 Dec 2022eess.AS cs.LG cs.SDcited by 19

Not on the map. Why?

Abstract (arXiv)

While human evaluation is the most reliable metric for evaluating speech generation systems, it is generally costly and time-consuming. Previous studies on automatic speech quality assessment address the problem by predicting human evaluation scores with machine learning models. However, they rely on supervised learning and thus suffer from high annotation costs and domain-shift problems. We propose SpeechLMScore, an unsupervised metric to evaluate generated speech using a speech-language model. SpeechLMScore computes the average log-probability of a speech signal by mapping it into discrete tokens and measures the average probability of generating the sequence of tokens. Therefore, it does not require human annotation and is a highly scalable framework. Evaluation results demonstrate that the proposed metric shows a promising correlation with human evaluation scores on different speech generation tasks including voice conversion, text-to-speech, and speech enhancement.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 25 of its 44 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

25 of 44 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  8. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  9. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  10. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  11. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  12. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
13 more
  1. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 235
  2. 2020The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Resultscited by 128
  3. 2019MOSNet: Deep Learning based Objective Assessment for Voice Conversioncited by 126
  4. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  5. 2020Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversioncited by 75
  6. 2021Generalization Ability of MOS Prediction Networkscited by 69
  7. 2018The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methodscited by 68
  8. 2019ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkitcited by 60
  9. 2022The VoiceMOS Challenge 2022cited by 57
  10. 2016AutoMOS: Learning a non-intrusive assessor of naturalness-of-speechcited by 22
  11. 2020DiscreTalk: Text-to-Speech as a Machine Translation Problemcited by 21
  12. 2022Speech Quality Assessment through MOS using Non-Matching Referencescited by 16
  13. 2022SQuId: Measuring Speech Naturalness in Many Languagescited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 8; 2025: 7; 2026: 1.

19 papers.

Sort:

By area:

  1. 2024Enabling Auditory Large Language Models for Automatic Speech Quality Evaluationcited by 13
  2. 2024MSLM-S2ST: A Multitask Speech Language Model for Textless Speech-to-Speech Translation with Speaker Style Preservationcited by 11
  3. 2025From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speechcited by 9
  4. 2024Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speechcited by 7
  5. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generationcited by 6
  6. 2024A Survey of Deep Learning Audio Generation Methodscited by 5
  7. 2023Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokenscited by 4
  8. 2023Uncertainty as a Predictor: Leveraging Self-Supervised Learning for Zero-Shot MOS Predictioncited by 4
  9. 2024TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languagescited by 4
  10. 2023Evaluating Speech Synthesis by Training Recognizers on Synthetic Speechcited by 3
  11. 2025Bridging The Multi-Modality Gaps of Audio, Visual and Linguistic for Speech Enhancementcited by 3
  12. 2024A Collection of Pragmatic-Similarity Judgments over Spoken Dialog Utterancescited by 2
7 more
  1. 2025Objective Evaluation of Prosody and Intelligibility in Speech Synthesis via Conditional Prediction of Discrete Tokenscited by 2
  2. 2024An Empirical Study of Speech Language Models for Prompt-Conditioned Speech Synthesiscited by 1
  3. 2024Non-intrusive Speech Quality Assessment with Diffusion Models Trained on Clean Speechcited by 1
  4. 2025SpeechDialogueFactory: Generating High-Quality Speech Dialogue Data to Accelerate Your Speech-LLM Developmentcited by 1
  5. 2025Improving Perceptual Audio Aesthetic Assessment via Triplet Loss and Self-Supervised Embeddingscited by 1
  6. 2025Model as Loss: A Self-Consistent Training Paradigmcited by 0
  7. 2026OmniJudge or OmniBias? Diagnosing Multimodal Judges through Balanced, Decoupled Lensescited by 0