mlsanityData to 19 Oct 2026

Benchmarking Neural Speech Codec Intelligibility with SITool

arXiv 2506.01731 ↗First posted 2 Jun 2025eess.AScited by 1

On the map: Speech and audio language models → Vocoders and speech synthesis

Abstract (arXiv)

Speech intelligibility assessment is essential for evaluating neural speech codecs, yet most evaluation efforts focus on overall quality rather than intelligibility. Only a few publicly available tools exist for conducting standardized intelligibility tests, like the Diagnostic Rhyme Test (DRT) and Modified Rhyme Test (MRT). We introduce the Speech Intelligibility Toolkit for Subjective Evaluation (SITool), a Flask-based web application for conducting DRT and MRT in laboratory and crowdsourcing settings. We use SITool to benchmark 13 neural and traditional speech codecs, analyzing phoneme-level degradations and comparing subjective DRT results with objective intelligibility metrics. Our findings show that, while neural speech codecs can outperform traditional ones in subjective intelligibility, only STOI and ESTOI - not WER - significantly correlate with subjective results, although they struggle to capture gender and wordlist-specific variations observed in subjective evaluations.

Key facts. Cited by 1 paper in the map (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 17 of its 37 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

17 of 37 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024SNAC: Multi-Scale Neural Audio Codec18 Oct
  2. 2024Low Bitrate High-Quality RVQGAN-based Discrete Speech Tokenizer10 Oct
  3. 2024Neural Speech Coding for Real-time Communications using Constant Bitrate Scalar Quantization14 May
  4. 2023Generative De-Quantization for Neural Speech Codec via Latent Diffusion14 Nov
  5. 2022Ultra-Low-Bitrate Speech Coding with Pretrained Transformers5 Jul
  6. 2021SoundStream: An End-to-End Neural Audio Codec7 Jul

Other references · 11

  1. 2017Neural Discrete Representation Learningcited by 4,843
  2. 2022High Fidelity Neural Audio Compressioncited by 917
  3. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 711
  4. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 619
  5. 2018LPCNet: Improving Neural Speech Synthesis Through Linear Predictioncited by 160
  6. 2020ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metriccited by 120
  7. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 84
  8. 2020StyleMelGAN: An Efficient High-Fidelity Adversarial Vocoder with Temporal Adaptive Normalizationcited by 41
  9. 2021Generative Speech Coding with Predictive Variance Regularizationcited by 30
  10. 2024Neural Speech and Audio Coding: Modern AI Technology Meets Traditional Codecscited by 9
  11. 2024Crowdsourced Multilingual Speech Intelligibility Testingcited by 6

Built on by

Cited by year (2026 to 19 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Assessing the Impact of Noise and Speech Enhancement on the Intelligibility of Speech Codecscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025From Evaluation to Optimization: Neural Speech Assessment for Downstream Applications ↗
  2. 2025Aligning ASR Evaluation with Human and LLM Judgments: Intelligibility Metrics Using Phonetic, Semantic, and NLI Approaches
  3. 2023Evaluating Speech Synthesis by Training Recognizers on Synthetic Speech
  4. 2026How Robust Are Neural Audio Codecs for African Speech? A Multi-Task Benchmark and the Limits of Perceptual Quality
  5. 2025SP-MCQA: Evaluating Intelligibility of TTS Beyond the Word Level
  6. 2025Evaluating Objective Speech Quality Metrics for Neural Audio Codecs
  7. 2022Prediction of speech intelligibility with DNN-based performance measures
  8. 2026PathBench: Speech Intelligibility Benchmark for Automatic Pathological Speech Assessment