# One Whisper to Grade Them All

arXiv: https://arxiv.org/abs/2507.17918 · HTML: https://mlsanity.com/paper/2507.17918/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 24 references that are in the map.
- First posted on arXiv 23 Jul 2025.

## Builds on (most cited first, 6 of 6)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Proficiency assessment of L2 spoken English using wav2vec 2.0](https://mlsanity.com/paper/2210.13168/) (2022)
- [Speak & Improve Corpus 2025: an L2 English Speech Corpus for Language Assessment and Feedback](https://mlsanity.com/paper/2412.11986/) (2024)
- [Speak & Improve Challenge 2025: Tasks and Baseline Systems](https://mlsanity.com/paper/2412.11985/) (2024)

## Built on by (most cited first, 2 of 2)

- [Bias Analysis of L2 Speaking Assessment Systems Using Concept Activation Vectors](https://mlsanity.com/paper/2608.06300/) (2026)
- [CASA: Content-Acoustic Speaking Assessment with Speech Encoder and Large Language Model](https://mlsanity.com/paper/2608.13101/) (2026)

## Abstract (arXiv)

We present an efficient end-to-end approach for holistic Automatic Speaking Assessment (ASA) of multi-part second-language tests, developed for the 2025 Speak & Improve Challenge. Our system's main novelty is the ability to process all four spoken responses with a single Whisper-small encoder, combine all information via a lightweight aggregator, and predict the final score. This architecture removes the need for transcription and per-part models, cuts inference time, and makes ASA practical for large-scale Computer-Assisted Language Learning systems. Our system achieved a Root Mean Squared Error (RMSE) of 0.384, outperforming the text-based baseline (0.44) while using at most 168M parameters (about 70% of Whisper-small). Furthermore, we propose a data sampling strategy, allowing the model to train on only 44.8% of the speakers in the corpus and still reach 0.383 RMSE, demonstrating improved performance on imbalanced classes and strong data efficiency.