# CantoASR: Prosody-Aware ASR-LALM Collaboration for Low-Resource Cantonese

arXiv: https://arxiv.org/abs/2511.04139 · HTML: https://mlsanity.com/paper/2511.04139/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 6 Nov 2025.

## Builds on (most cited first, 23 of 23)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting](https://mlsanity.com/paper/2309.15649/) (2023)
- [Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks](https://mlsanity.com/paper/2411.05361/) (2024)
- [DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment](https://mlsanity.com/paper/2507.02768/) (2025)
- [ByT5 model for massively multilingual grapheme-to-phoneme conversion](https://mlsanity.com/paper/2204.03067/) (2022)
- [N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space](https://mlsanity.com/paper/2303.00456/) (2023)
- [Towards audio language modeling -- an overview](https://mlsanity.com/paper/2402.13236/) (2024)
- [ASR Error Correction using Large Language Models](https://mlsanity.com/paper/2409.09554/) (2024)
- [Building a Taiwanese Mandarin Spoken Language Model: A First Attempt](https://mlsanity.com/paper/2411.07111/) (2024)
- [BART based semantic correction for Mandarin automatic speech recognition system](https://mlsanity.com/paper/2104.05507/) (2021)
- [Hey ASR System! Why Aren't You More Inclusive? Automatic Speech Recognition Systems' Bias and Proposed Bias Mitigation Techniques. A Literature Review](https://mlsanity.com/paper/2211.09511/) (2022)
- [Automatic Speech Recognition Datasets in Cantonese: A Survey and New Dataset](https://mlsanity.com/paper/2201.02419/) (2022)
- [Improving Speech Emotion Recognition in Under-Resourced Languages via Speech-to-Speech Translation with Bootstrapping Data Selection](https://mlsanity.com/paper/2409.10985/) (2024)
- [HKCanto-Eval: A Benchmark for Evaluating Cantonese Language Understanding and Cultural Comprehension in LLMs](https://mlsanity.com/paper/2503.12440/) (2025)
- [Fewer Hallucinations, More Verification: A Three-Stage LLM-Based Framework for ASR Error Correction](https://mlsanity.com/paper/2505.24347/) (2025)
- [Developing a Multilingual Dataset and Evaluation Metrics for Code-Switching: A Focus on Hong Kong's Polylingual Dynamics](https://mlsanity.com/paper/2310.17953/) (2023)
- [Full-text Error Correction for Chinese Speech Recognition with Large Language Model](https://mlsanity.com/paper/2409.07790/) (2024)
- [Boosting Chinese ASR Error Correction with Dynamic Error Scaling Mechanism](https://mlsanity.com/paper/2308.03423/) (2023)

## Built on by (most cited first, 3 of 3)

- [Now You Hear Me: Audio Narrative Attacks Against Large Audio-Language Models](https://mlsanity.com/paper/2601.23255/) (2026)
- [VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech](https://mlsanity.com/paper/2604.17248/) (2026)
- [Efficiently Adapting Spoken Language Models for the Singaporean Context](https://mlsanity.com/paper/2607.10092/) (2026)

## Abstract (arXiv)

Automatic speech recognition (ASR) is critical for language accessibility, yet low-resource Cantonese remains challenging due to limited annotated data, six lexical tones, tone sandhi, and accent variation. Existing ASR models, such as Whisper, often suffer from high word error rates. Large audio-language models (LALMs), in contrast, can leverage broader contextual reasoning but still require explicit tonal and prosodic acoustic cues. We introduce CantoASR, a collaborative ASR-LALM error correction framework that integrates forced alignment for acoustic feature extraction, a LoRA-finetuned Whisper for improved tone discrimination, and an instruction-tuned Qwen-Audio for prosody-aware correction. Evaluations on spontaneous Cantonese data show substantial CER gains over Whisper-Large-V3. These findings suggest that integrating acoustic cues with LALM reasoning provides a scalable strategy for low-resource tonal and dialectal ASR.