mlsanityData to 14 Oct 2026

Do Large Language Models Grasp The Grammar? Evidence from Grammar-Book-Guided Probing in Luxembourgish

arXiv 2510.24856 ↗First posted 28 Oct 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Grammar refers to the system of rules that governs the structural organization and the semantic relations among linguistic units such as sentences, phrases, and words within a given language. In natural language processing, there remains a notable scarcity of grammar focused evaluation protocols, a gap that is even more pronounced for low-resource languages. Moreover, the extent to which large language models genuinely comprehend grammatical structure, especially the mapping between syntactic structures and meanings, remains under debate. To investigate this issue, we propose a Grammar Book Guided evaluation pipeline intended to provide a systematic and generalizable framework for grammar evaluation consisting of four key stages, and in this work we take Luxembourgish as a case study. The results show a weak positive correlation between translation performance and grammatical understanding, indicating that strong translations do not necessarily imply deep grammatical competence. Larger models perform well overall due to their semantic strength but remain weak in morphology and syntax, struggling particularly with Minimal Pair tasks, while strong reasoning ability offers a promising way to enhance their grammatical understanding.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 24 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

15 of 24 references.

References · 15

  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  2. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  3. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  4. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  5. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 75
  6. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  7. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 46
  8. 2024Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languagescited by 38
  9. 2020Curious Case of Language Generation Evaluation Metrics: A Cautionary Talecited by 28
  10. 2024Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Researchcited by 27
  11. 2024Can LLMs Really Learn to Translate a Low-Resource Language from One Grammar Book?cited by 23
  12. 2024How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgmentscited by 14
3 more
  1. 2024Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Datacited by 7
  2. 2025How Syntax Specialization Emerges in Language Modelscited by 7
  3. 2025MiLiC-Eval: Benchmarking Multilingual LLMs for China's Minority Languagescited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Read it in Two Steps: Translating Extremely Low-Resource Languages with Code-Augmented Grammar Books
  2. 2024LLMs' Understanding of Natural Language Revealed
  3. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modeling
  4. 2025CPG-EVAL: A Multi-Tiered Benchmark for Evaluating the Chinese Pedagogical Grammar Competence of Large Language Models
  5. 2022Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding
  6. 2025Towards Corpus-Grounded Agentic LLMs for Multilingual Grammatical Analysis
  7. 2024Language models align with human judgments on key grammatical constructions
  8. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning