# How Useful are LLMs for Grammar Engineering? Cantonese ParGram Resources and Controlled Experimental Evaluation with English Baselines

arXiv: https://arxiv.org/abs/2608.23448 · HTML: https://mlsanity.com/paper/2608.23448/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 37 references that are in the map.
- First posted on arXiv 24 Aug 2026.

## Builds on (most cited first, 4 of 4)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Can LLMs Help Create Grammar?: Automating Grammar Creation for Endangered Languages with In-Context Learning](https://mlsanity.com/paper/2412.10960/) (2024)
- [GaelEval: Benchmarking LLM Performance for Scottish Gaelic](https://mlsanity.com/paper/2604.02135/) (2026)
- [Grammar Engineering Meets LLMs: Development of Cantonese and Irish ParGram Treebanks](https://mlsanity.com/paper/2608.07283/) (2026)

## Abstract (arXiv)

This paper presents new Cantonese ParGram resources and evaluates LLMs for knowledge-driven grammar engineering within a controlled experimental paradigm. Using Cantonese ParGram resources as gold standards, with corresponding English baselines, we investigate whether OpenAI's gpt-oss-120b and GPT-5.4 can generate machine-processable grammars from sentences and target formal structures under systematically varied prompting conditions. GPT-5.4 outperformed gpt-oss-120b, while grammars generated from target formal structures generally outperformed those generated from sentences. Although both models could generate locally plausible phrase-structure rules, lexical entries, and templates, they often struggled to coordinate interacting formal constraints, especially in multi-construction settings. The results characterize both the capabilities and limitations of current LLMs for potential integration into AI-assisted expert workflows: LLMs may support intermediate stages of grammar development, but human linguistic expertise remains central to analysis, validation, and refinement. The study also contributes new Cantonese symbolic grammatical resources.