# A Study on Incorporating Whisper for Robust Speech Assessment

arXiv: https://arxiv.org/abs/2309.12766 · HTML: https://mlsanity.com/paper/2309.12766/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2023.

## Built on by (most cited first, 11 of 11)

- [DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment](https://mlsanity.com/paper/2507.02768/) (2025)
- [AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation](https://mlsanity.com/paper/2507.12705/) (2025)
- [A Layer-Anchoring Strategy for Enhancing Cross-Lingual Speech Emotion Recognition](https://mlsanity.com/paper/2407.04966/) (2024)
- [From Scores to Preferences: Redefining MOS Benchmarking for Speech Quality Reward Modeling](https://mlsanity.com/paper/2510.00743/) (2025)
- [Bridging The Multi-Modality Gaps of Audio, Visual and Linguistic for Speech Enhancement](https://mlsanity.com/paper/2501.13375/) (2025)
- [SVSNet+: Enhancing Speaker Voice Similarity Assessment Models with Representations from Speech Foundation Models](https://mlsanity.com/paper/2406.08445/) (2024)
- [Automated evaluation of children's speech fluency for low-resource languages](https://mlsanity.com/paper/2505.19671/) (2025)
- [Improving Perceptual Audio Aesthetic Assessment via Triplet Loss and Self-Supervised Embeddings](https://mlsanity.com/paper/2509.03292/) (2025)
- [Improving Robustness of Diffusion-Based Zero-Shot Speech Synthesis via Stable Formant Generation](https://mlsanity.com/paper/2409.09311/) (2024)
- [MedVoiceBias: A Controlled Study of Audio LLM Behavior in Clinical Decision-Making](https://mlsanity.com/paper/2511.06592/) (2025)
- [CAL-MOS: Bridging Layers with Adapters for Robust MOS Prediction Across Speech Foundation Models](https://mlsanity.com/paper/2609.14956/) (2026)

## Abstract (arXiv)

This research introduces an enhanced version of the multi-objective speech assessment model--MOSA-Net+, by leveraging the acoustic features from Whisper, a large-scaled weakly supervised model. We first investigate the effectiveness of Whisper in deploying a more robust speech assessment model. After that, we explore combining representations from Whisper and SSL models. The experimental results reveal that Whisper's embedding features can contribute to more accurate prediction performance. Moreover, combining the embedding features from Whisper and SSL models only leads to marginal improvement. As compared to intrusive methods, MOSA-Net, and other SSL-based speech assessment models, MOSA-Net+ yields notable improvements in estimating subjective quality and intelligibility scores across all evaluation metrics in Taiwan Mandarin Hearing In Noise test - Quality & Intelligibility (TMHINT-QI) dataset. To further validate its robustness, MOSA-Net+ was tested in the noisy-and-enhanced track of the VoiceMOS Challenge 2023, where it obtained the top-ranked performance among nine systems.