# Current LLMs still cannot 'talk much' about grammar modules: Evidence from syntax

arXiv: https://arxiv.org/abs/2603.20114 · HTML: https://mlsanity.com/paper/2603.20114/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 54 references that are in the map.
- First posted on arXiv 20 Mar 2026.

## Builds on (most cited first, 8 of 8)

- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [Syntactic Structure from Deep Learning](https://mlsanity.com/paper/2004.10827/) (2020)
- [Machine Translation from Signed to Spoken Languages: State of the Art and Challenges](https://mlsanity.com/paper/2202.03086/) (2022)
- [Generative linguistics contribution to artificial intelligence: Where this contribution lies?](https://mlsanity.com/paper/2410.20221/) (2024)
- [What fifty-one years of Linguistics and Artificial Intelligence research tell us about their correlation: A scientometric analysis](https://mlsanity.com/paper/2411.19858/) (2024)
- [Artificial intelligence contribution to translation industry: looking back and forward](https://mlsanity.com/paper/2411.19855/) (2024)
- [Can ChatGPT capture swearing nuances? Evidence from translating Arabic oaths](https://mlsanity.com/paper/2412.02466/) (2024)

## Abstract (arXiv)

We aim to examine the extent to which Large Language Models (LLMs) can 'talk much' about grammar modules, providing evidence from syntax core properties translated by ChatGPT into Arabic. We collected 44 terms from generative syntax previous works, including books and journal articles, as well as from our experience in the field. These terms were translated by humans, and then by ChatGPT-5. We then analyzed and compared both translations. We used an analytical and comparative approach in our analysis. Findings unveil that LLMs still cannot 'talk much' about the core syntax properties embedded in the terms under study involving several syntactic and semantic challenges: only 25% of ChatGPT translations were accurate, while 38.6% were inaccurate, and 36.4.% were partially correct, which we consider appropriate. Based on these findings, a set of actionable strategies were proposed, the most notable of which is a close collaboration between AI specialists and linguists to better LLMs' working mechanism for accurate or at least appropriate translation.