# How Well Do LLMs Handle Cantonese? Benchmarking Cantonese Capabilities of Large Language Models

arXiv: https://arxiv.org/abs/2408.16756 · HTML: https://mlsanity.com/paper/2408.16756/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 68 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [Syntactic Data Augmentation Increases Robustness to Inference Heuristics](https://mlsanity.com/paper/2004.11999/) (2020)
- [Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation](https://mlsanity.com/paper/1905.05621/) (2019)
- [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022)
- [IMaT: Unsupervised Text Attribute Transfer via Iterative Matching and Translation](https://mlsanity.com/paper/1901.11333/) (2019)
- [Substructure Substitution: Structured Data Augmentation for NLP](https://mlsanity.com/paper/2101.00411/) (2021)
- [LoRA Meets Dropout under a Unified Framework](https://mlsanity.com/paper/2403.00812/) (2024)
- [SupCL-Seq: Supervised Contrastive Learning for Downstream Optimized Sequence Representations](https://mlsanity.com/paper/2109.07424/) (2021)

## Built on by (most cited first, 3 of 3)

- [TRACE: Trajectory-Aware Comprehensive Evaluation for Deep Research Agents](https://mlsanity.com/paper/2602.21230/) (2026)
- [Transport and Merge: Cross-Architecture Merging for Large Language Models](https://mlsanity.com/paper/2602.05495/) (2026)
- [Probing Cultural Awareness in LLMs: A Case Study of Cross-Culture Aesthetic Stylistics](https://mlsanity.com/paper/2605.27296/) (2026)

## Abstract (arXiv)

The rapid evolution of large language models (LLMs) has transformed the competitive landscape in natural language processing (NLP), particularly for English and other data-rich languages. However, underrepresented languages like Cantonese, spoken by over 85 million people, face significant development gaps, which is particularly concerning given the economic significance of the Guangdong-Hong Kong-Macau Greater Bay Area, and in substantial Cantonese-speaking populations in places like Singapore and North America. Despite its wide use, Cantonese has scant representation in NLP research, especially compared to other languages from similarly developed regions. To bridge these gaps, we outline current Cantonese NLP methods and introduce new benchmarks designed to evaluate LLM performance in factual generation, mathematical logic, complex reasoning, and general knowledge in Cantonese, which aim to advance open-source Cantonese LLM technology. We also propose future research directions and recommended models to enhance Cantonese LLM development.