# A Finetuned SpeechLLM for Joint Multi-Granular L2 Assessment and Natural-Language Rationales

arXiv: https://arxiv.org/abs/2606.09470 · HTML: https://mlsanity.com/paper/2606.09470/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 8 Jun 2026.

## Builds on (most cited first, 28 of 28)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities](https://mlsanity.com/paper/2406.11768/) (2024)
- [On decoder-only architecture for speech-to-text and large language model integration](https://mlsanity.com/paper/2307.03917/) (2023)
- [Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities](https://mlsanity.com/paper/2503.03983/) (2025)
- [Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models](https://mlsanity.com/paper/2402.04614/) (2024)
- [speechocean762: An Open-Source Non-native English Speech Corpus For Pronunciation Assessment](https://mlsanity.com/paper/2104.01378/) (2021)
- [Transformer-Based Multi-Aspect Multi-Granularity Non-Native English Speaker Pronunciation Assessment](https://mlsanity.com/paper/2205.03432/) (2022)
- [Automatic Pronunciation Assessment using Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2204.03863/) (2022)
- [Rethinking DPO: The Role of Rejected Responses in Preference Misalignment](https://mlsanity.com/paper/2506.12725/) (2025)
- [Assessment of L2 Oral Proficiency using Speech Large Language Models](https://mlsanity.com/paper/2505.21148/) (2025)
- [SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation](https://mlsanity.com/paper/2510.14664/) (2025)
- [A Hierarchical Context-aware Modeling Approach for Multi-aspect and Multi-granular Pronunciation Assessment](https://mlsanity.com/paper/2305.18146/) (2023)
- [Evaluating Human Alignment and Model Faithfulness of LLM Rationale](https://mlsanity.com/paper/2407.00219/) (2024)
- [Read to Hear: A Zero-Shot Pronunciation Assessment Using Textual Descriptions and LLMs](https://mlsanity.com/paper/2509.14187/) (2025)
- [Exploring the Potential of Large Multimodal Models as Effective Alternatives for Pronunciation Assessment](https://mlsanity.com/paper/2503.11229/) (2025)
- [Enhancing GOP in CTC-Based Mispronunciation Detection with Phonological Knowledge](https://mlsanity.com/paper/2506.02080/) (2025)
- [Zero-Shot Speech LLMs for Multi-Aspect Evaluation of L2 Speech: Challenges and Opportunities](https://mlsanity.com/paper/2601.16230/) (2026)
- [Rubric-Guided Fine-tuning of SpeechLLMs for Multi-Aspect, Multi-Rater L2 Reading-Speech Assessment](https://mlsanity.com/paper/2603.16889/) (2026)
- [Evaluating Logit-Based GOP Scores for Mispronunciation Detection](https://mlsanity.com/paper/2506.12067/) (2025)
- [Fine-Tuning Large Multimodal Models for Automatic Pronunciation Assessment](https://mlsanity.com/paper/2509.15701/) (2025)

## Built on by (most cited first, 1 of 1)

- [Interpretable, Fairly Evaluated Automated L2 Speaking Assessment that Beats the Single-Human Ceiling and Why Pause Encoding Does Not Change LLM Fluency Scores](https://mlsanity.com/paper/2608.26137/) (2026)

## Abstract (arXiv)

Automated L2 speech assessment can assign proficiency labels, but often lacks interpretability. We propose a rubric-guided SpeechLLM for multi-aspect, multi-granular assessment, trained with a hybrid objective combining supervised fine-tuning and Bounded Direct Preference Optimization. The model jointly predicts ordinal labels at the sentence-level (accuracy, fluency, prosody), word/phoneme-level accuracy, and generates a natural-language rationale in the same response. On SpeechOcean762, our approach matches or outperforms single-granularity models while remaining competitive with prior approaches. We analyze rationale reliability along two axes: self-consistency with model predictions and alignment with ground-truth labels, using sentiment consistency (plausibility) and mention-based agreement (faithfulness). Rationales are plausible at the sentence level, but faithfulness degrades at the word/phoneme level: references are sparse and weakly aligned with token-level labels.