# An ASR-free Fluency Scoring Approach with Self-Supervised Learning

arXiv: https://arxiv.org/abs/2302.09928 · HTML: https://mlsanity.com/paper/2302.09928/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Feb 2023.

## Built on by (most cited first, 6 of 6)

- [MultiPA: A Multi-task Speech Pronunciation Assessment Model for Open Response Scenarios](https://mlsanity.com/paper/2308.12490/) (2023)
- [Pronunciation Assessment with Multi-modal Large Language Models](https://mlsanity.com/paper/2407.09209/) (2024)
- [Phonetic and Prosody-aware Self-supervised Learning Approach for Non-native Fluency Scoring](https://mlsanity.com/paper/2305.11438/) (2023)
- [Preserving Phonemic Distinctions for Ordinal Regression: A Novel Loss Function for Automatic Pronunciation Assessment](https://mlsanity.com/paper/2310.01839/) (2023)
- [CBF-AFA: Chunk-Based Multi-SSL Fusion for Automatic Fluency Assessment](https://mlsanity.com/paper/2506.20243/) (2025)
- [Interpretable, Fairly Evaluated Automated L2 Speaking Assessment that Beats the Single-Human Ceiling and Why Pause Encoding Does Not Change LLM Fluency Scores](https://mlsanity.com/paper/2608.26137/) (2026)

## Abstract (arXiv)

A typical fluency scoring system generally relies on an automatic speech recognition (ASR) system to obtain time stamps in input speech for either the subsequent calculation of fluency-related features or directly modeling speech fluency with an end-to-end approach. This paper describes a novel ASR-free approach for automatic fluency assessment using self-supervised learning (SSL). Specifically, wav2vec2.0 is used to extract frame-level speech features, followed by K-means clustering to assign a pseudo label (cluster index) to each frame. A BLSTM-based model is trained to predict an utterance-level fluency score from frame-level SSL features and the corresponding cluster indexes. Neither speech transcription nor time stamp information is required in the proposed system. It is ASR-free and can potentially avoid the ASR errors effect in practice. Experimental results carried out on non-native English databases show that the proposed approach significantly improves the performance in the "open response" scenario as compared to previous methods and matches the recently reported performance in the "read aloud" scenario.