# How well can LLMs Grade Essays in Arabic?

arXiv: https://arxiv.org/abs/2501.16516 · HTML: https://mlsanity.com/paper/2501.16516/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 43 references that are in the map.
- First posted on arXiv 27 Jan 2025.

## Builds on (most cited first, 14 of 14)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://mlsanity.com/paper/2304.01852/) (2023)
- [Few-shot training LLMs for project-specific code-summarization](https://mlsanity.com/paper/2207.04237/) (2022)
- [AceGPT, Localizing Large Language Models in Arabic](https://mlsanity.com/paper/2309.12053/) (2023)
- [Fine-tuning ChatGPT for Automatic Scoring](https://mlsanity.com/paper/2310.10072/) (2023)
- [Can Large Language Models Automatically Score Proficiency of Written Essays?](https://mlsanity.com/paper/2403.06149/) (2024)
- [Masader: Metadata Sourcing for Arabic Text and Speech Data Resources](https://mlsanity.com/paper/2110.06744/) (2021)
- [Beyond English: Evaluating LLMs for Arabic Grammatical Error Correction](https://mlsanity.com/paper/2312.08400/) (2023)
- [Can Large Language Models Make the Grade? An Empirical Study Evaluating LLMs Ability to Mark Short Answer Questions in K-12 Education](https://mlsanity.com/paper/2405.02985/) (2024)
- [Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability](https://mlsanity.com/paper/2401.05655/) (2024)
- [Prediction of Arabic Legal Rulings using Large Language Models](https://mlsanity.com/paper/2310.10260/) (2023)
- [An Automated System for Essay Scoring of Online Exams in Arabic based on Stemming Techniques and Levenshtein Edit Operations](https://mlsanity.com/paper/1611.02815/) (2016)

## Built on by (most cited first, 6 of 6)

- [LAILA: A Large Trait-Based Dataset for Arabic Automated Essay Scoring](https://mlsanity.com/paper/2512.24235/) (2025)
- [Agreement Between Large Language Models and Human Raters in Essay Scoring: A Research Synthesis](https://mlsanity.com/paper/2512.14561/) (2025)
- [Qayyem: A Real-time Platform for Scoring Proficiency of Arabic Essays](https://mlsanity.com/paper/2603.01009/) (2026)
- [Structured Prompting for Arabic Essay Proficiency: A Trait-Centric Evaluation Approach](https://mlsanity.com/paper/2603.19668/) (2026)
- [Automated Scoring of Arabic Text Using Large Language Models: A Literature Review](https://mlsanity.com/paper/2606.09830/) (2026)
- [The Order Matters: Sequential Fine-Tuning of LLaMA for Coherent Automated Essay Scoring](https://mlsanity.com/paper/2606.10327/) (2026)

## Abstract (arXiv)

This research assesses the effectiveness of state-of-the-art large language models (LLMs), including ChatGPT, Llama, Aya, Jais, and ACEGPT, in the task of Arabic automated essay scoring (AES) using the AR-AES dataset. It explores various evaluation methodologies, including zero-shot, few-shot in-context learning, and fine-tuning, and examines the influence of instruction-following capabilities through the inclusion of marking guidelines within the prompts. A mixed-language prompting strategy, integrating English prompts with Arabic content, was implemented to improve model comprehension and performance. Among the models tested, ACEGPT demonstrated the strongest performance across the dataset, achieving a Quadratic Weighted Kappa (QWK) of 0.67, but was outperformed by a smaller BERT-based model with a QWK of 0.88. The study identifies challenges faced by LLMs in processing Arabic, including tokenization complexities and higher computational demands. Performance variation across different courses underscores the need for adaptive models capable of handling diverse assessment formats and highlights the positive impact of effective prompt engineering on improving LLM outputs. To the best of our knowledge, this study is the first to empirically evaluate the performance of multiple generative Large Language Models (LLMs) on Arabic essays using authentic student data.