# Benchmarking Neural Speech Codec Intelligibility with SITool

arXiv: https://arxiv.org/abs/2506.01731 · HTML: https://mlsanity.com/paper/2506.01731/

## Key facts

- Cited by 1 papers in the map (data through 19 Oct 2026).
- Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models.
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Placed in

- Speech and audio language models / [Vocoders and speech synthesis](https://mlsanity.com/speech/vocoders-and-speech-synthesis/)

## Builds on (most cited first, 17 of 17)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [LPCNet: Improving Neural Speech Synthesis Through Linear Prediction](https://mlsanity.com/paper/1810.11846/) (2018)
- [ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric](https://mlsanity.com/paper/2004.09584/) (2020)
- [SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound](https://mlsanity.com/paper/2405.00233/) (2024)
- [SNAC: Multi-Scale Neural Audio Codec](https://mlsanity.com/paper/2410.14411/) (2024)
- [StyleMelGAN: An Efficient High-Fidelity Adversarial Vocoder with Temporal Adaptive Normalization](https://mlsanity.com/paper/2011.01557/) (2020)
- [Generative Speech Coding with Predictive Variance Regularization](https://mlsanity.com/paper/2102.09660/) (2021)
- [Generative De-Quantization for Neural Speech Codec via Latent Diffusion](https://mlsanity.com/paper/2311.08330/) (2023)
- [Ultra-Low-Bitrate Speech Coding with Pretrained Transformers](https://mlsanity.com/paper/2207.02262/) (2022)
- [Neural Speech and Audio Coding: Modern AI Technology Meets Traditional Codecs](https://mlsanity.com/paper/2408.06954/) (2024)
- [Crowdsourced Multilingual Speech Intelligibility Testing](https://mlsanity.com/paper/2403.14817/) (2024)
- [Neural Speech Coding for Real-time Communications using Constant Bitrate Scalar Quantization](https://mlsanity.com/paper/2405.08417/) (2024)
- [Low Bitrate High-Quality RVQGAN-based Discrete Speech Tokenizer](https://mlsanity.com/paper/2410.08325/) (2024)

## Built on by (most cited first, 1 of 1)

- [Assessing the Impact of Noise and Speech Enhancement on the Intelligibility of Speech Codecs](https://mlsanity.com/paper/2605.03776/) (2026)

## Abstract (arXiv)

Speech intelligibility assessment is essential for evaluating neural speech codecs, yet most evaluation efforts focus on overall quality rather than intelligibility. Only a few publicly available tools exist for conducting standardized intelligibility tests, like the Diagnostic Rhyme Test (DRT) and Modified Rhyme Test (MRT). We introduce the Speech Intelligibility Toolkit for Subjective Evaluation (SITool), a Flask-based web application for conducting DRT and MRT in laboratory and crowdsourcing settings. We use SITool to benchmark 13 neural and traditional speech codecs, analyzing phoneme-level degradations and comparing subjective DRT results with objective intelligibility metrics. Our findings show that, while neural speech codecs can outperform traditional ones in subjective intelligibility, only STOI and ESTOI - not WER - significantly correlate with subjective results, although they struggle to capture gender and wordlist-specific variations observed in subjective evaluations.