Fine-Tuning Large Multimodal Models for Automatic Pronunciation Assessment
Not on the map. Why?
Abstract (arXiv)
Automatic Pronunciation Assessment (APA) is critical for Computer-Assisted Language Learning (CALL), requiring evaluation across multiple granularities and aspects. Large Multimodal Models (LMMs) present new opportunities for APA, but their effectiveness in fine-grained assessment remains uncertain. This work investigates fine-tuning LMMs for APA using the Speechocean762 dataset and a private corpus. Fine-tuning significantly outperforms zero-shot settings and achieves competitive results on single-granularity tasks compared to public and commercial systems. The model performs well at word and sentence levels, while phoneme-level assessment remains challenging. We also observe that the Pearson Correlation Coefficient (PCC) reaches 0.9, whereas Spearman's rank Correlation Coefficient (SCC) remains around 0.6, suggesting that SCC better reflects ordinal consistency. These findings highlight both the promise and limitations of LMMs for APA and point to future work on fine-grained modeling and rank-aware evaluation.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 16 of its 28 references that are in the map. First posted on arXiv 19 Sep 2025.
Builds on
16 of 28 references.
References · 16
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
- 2024Qwen2-Audio Technical Reportcited by 560
- 2021speechocean762: An Open-Source Non-native English Speech Corpus For Pronunciation Assessmentcited by 58
- 2022Transformer-Based Multi-Aspect Multi-Granularity Non-Native English Speaker Pronunciation Assessmentcited by 37
- 2025Assessment of L2 Oral Proficiency using Speech Large Language Modelscited by 14
- 2022Hierarchical Pronunciation Assessment with Multi-Aspect Attentioncited by 13
- 2023A Hierarchical Context-aware Modeling Approach for Multi-aspect and Multi-granular Pronunciation Assessmentcited by 13
- 2021An Approach to Mispronunciation Detection and Diagnosis with Acoustic, Phonetic and Linguistic (APL) Embeddingscited by 8
- 2023Automatic Pronunciation Assessment -- A Reviewcited by 7
- 2025Towards Efficient and Multifaceted Computer-assisted Pronunciation Training Leveraging Hierarchical Selective State Space Model and Decoupled Cross-entropy Losscited by 6
4 more
- 2023Assessing Phrase Break of ESL Speech with Pre-trained Language Models and Large Language Modelscited by 5
- 2024Pronunciation Assessment with Multi-modal Large Language Modelscited by 4
- 2025Exploring the Potential of Large Multimodal Models as Effective Alternatives for Pronunciation Assessmentcited by 4
- 2025English Pronunciation Evaluation without Complex Joint Training: LoRA Fine-tuned Speech Multimodal LLMcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025JCAPT: A Joint Modeling Approach for CAPT
- 2026CASA: Content-Acoustic Speaking Assessment with Speech Encoder and Large Language Model
- 2025Beyond Modality Limitations: A Unified MLLM Approach to Automated Speaking Assessment with Effective Curriculum Learning
- 2026LOPA: Enhancing Spoken Language Assessment via Latent Ordinal Prototype Alignment
- 2025AHELM: A Holistic Evaluation of Audio-Language Models
- 2026Efficiently Adapting Spoken Language Models for the Singaporean Context
- 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark
- 2024HDMoLE: Mixture of LoRA Experts with Hierarchical Routing and Dynamic Thresholds for Fine-Tuning LLM-based ASR Models