mlsanityData to 7 Oct 2026

Interpretable, Fairly Evaluated Automated L2 Speaking Assessment that Beats the Single-Human Ceiling and Why Pause Encoding Does Not Change LLM Fluency Scores

arXiv 2608.26137 ↗First posted 25 Jun 2026cs.CL cs.LG cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

Second-language (L2) English learners can rarely rehearse speaking with a partner. Speaking is also the most anxiety-laden skill. These gaps drive a fast-growing market for automated speaking practice and scoring. But an automated score is trustworthy only if it is accurate, interpretable, fair, and benchmarked against the right human bar. We build an interpretable feature-plus-LLM hybrid for spontaneous L2 dialogue. We evaluate it without ever fitting to the human labels, against the ICNALE Global Rating Archive: 140 speeches rated by ~80 trained raters on 10 analytic criteria. We score the 130 L2 speeches with usable audio. A deterministic De-Jong speech-timing composite reaches rho=0.764. Blended with a single text-LLM fluency judgment, it reaches Spearman rho=0.818 against the consensus gold. This agrees with the consensus better than 81% of the 80 individual trained raters: above the median rater (rho=0.73) and near the best, and at ~83% of the reliability-corrected maximum (kappa_max=0.99). The blend improves on the composite alone by +0.054 (paired-bootstrap 95% CI [0.017, 0.108], excludes 0); the LLM adds a coarse fluency ranking that the continuous composite refines. We also report a controlled null on pause encoding, bounded to effects below about +/-0.1 rho at this sample size. Holding the LLM and learner words fixed and varying only how pauses are written into the prompt, inline pause locations do not beat aggregate pause statistics (-0.069, CI [-0.15, +0.08]), and a grounded mid-clause criterion gives no reliable gain. The fluency signal comes from the measured speech-timing features, not from how pauses are written for the LLM. We back every claim with two agreeing learner-isolation methods, paired-bootstrap CIs, a monologue negative control, per-feature reproduction of classical measurements, and a per-L1 fairness audit.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 31 references that are in the map. First posted on arXiv 25 Jun 2026.

Builds on

15 of 31 references.

References · 15

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 66
  6. 2024Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuancescited by 27
  7. 2022Proficiency assessment of L2 spoken English using wav2vec 2.0cited by 20
  8. 2025Assessment of L2 Oral Proficiency using Speech Large Language Modelscited by 14
  9. 2023An ASR-free Fluency Scoring Approach with Self-Supervised Learningcited by 6
  10. 2025Natural Language-based Assessment of L2 Oral Proficiency using LLMscited by 6
  11. 2025Read to Hear: A Zero-Shot Pronunciation Assessment Using Textual Descriptions and LLMscited by 4
  12. 2026Zero-Shot Speech LLMs for Multi-Aspect Evaluation of L2 Speech: Challenges and Opportunitiescited by 2
3 more
  1. 2026Rubric-Guided Fine-tuning of SpeechLLMs for Multi-Aspect, Multi-Rater L2 Reading-Speech Assessmentcited by 2
  2. 2026Has Automated Essay Scoring Reached Sufficient Accuracy? Deriving Achievable QWK Ceilings from Classical Test Theorycited by 2
  3. 2026A Finetuned SpeechLLM for Joint Multi-Granular L2 Assessment and Natural-Language Rationalescited by 1

Built on by

Not cited yet.