# Fine-Tuning Large Multimodal Models for Automatic Pronunciation Assessment

arXiv: https://arxiv.org/abs/2509.15701 · HTML: https://mlsanity.com/paper/2509.15701/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 19 Sep 2025.

## Builds on (most cited first, 16 of 16)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [speechocean762: An Open-Source Non-native English Speech Corpus For Pronunciation Assessment](https://mlsanity.com/paper/2104.01378/) (2021)
- [Transformer-Based Multi-Aspect Multi-Granularity Non-Native English Speaker Pronunciation Assessment](https://mlsanity.com/paper/2205.03432/) (2022)
- [Assessment of L2 Oral Proficiency using Speech Large Language Models](https://mlsanity.com/paper/2505.21148/) (2025)
- [Hierarchical Pronunciation Assessment with Multi-Aspect Attention](https://mlsanity.com/paper/2211.08102/) (2022)
- [A Hierarchical Context-aware Modeling Approach for Multi-aspect and Multi-granular Pronunciation Assessment](https://mlsanity.com/paper/2305.18146/) (2023)
- [An Approach to Mispronunciation Detection and Diagnosis with Acoustic, Phonetic and Linguistic (APL) Embeddings](https://mlsanity.com/paper/2110.07274/) (2021)
- [Automatic Pronunciation Assessment -- A Review](https://mlsanity.com/paper/2310.13974/) (2023)
- [Towards Efficient and Multifaceted Computer-assisted Pronunciation Training Leveraging Hierarchical Selective State Space Model and Decoupled Cross-entropy Loss](https://mlsanity.com/paper/2502.07575/) (2025)
- [Assessing Phrase Break of ESL Speech with Pre-trained Language Models and Large Language Models](https://mlsanity.com/paper/2306.04980/) (2023)
- [Pronunciation Assessment with Multi-modal Large Language Models](https://mlsanity.com/paper/2407.09209/) (2024)
- [Exploring the Potential of Large Multimodal Models as Effective Alternatives for Pronunciation Assessment](https://mlsanity.com/paper/2503.11229/) (2025)
- [English Pronunciation Evaluation without Complex Joint Training: LoRA Fine-tuned Speech Multimodal LLM](https://mlsanity.com/paper/2509.02915/) (2025)

## Built on by (most cited first, 1 of 1)

- [A Finetuned SpeechLLM for Joint Multi-Granular L2 Assessment and Natural-Language Rationales](https://mlsanity.com/paper/2606.09470/) (2026)

## Abstract (arXiv)

Automatic Pronunciation Assessment (APA) is critical for Computer-Assisted Language Learning (CALL), requiring evaluation across multiple granularities and aspects. Large Multimodal Models (LMMs) present new opportunities for APA, but their effectiveness in fine-grained assessment remains uncertain. This work investigates fine-tuning LMMs for APA using the Speechocean762 dataset and a private corpus. Fine-tuning significantly outperforms zero-shot settings and achieves competitive results on single-granularity tasks compared to public and commercial systems. The model performs well at word and sentence levels, while phoneme-level assessment remains challenging. We also observe that the Pearson Correlation Coefficient (PCC) reaches 0.9, whereas Spearman's rank Correlation Coefficient (SCC) remains around 0.6, suggesting that SCC better reflects ordinal consistency. These findings highlight both the promise and limitations of LMMs for APA and point to future work on fine-grained modeling and rank-aware evaluation.