# LLMs can Perform Multi-Dimensional Analytic Writing Assessments: A Case Study of L2 Graduate-Level Academic English Writing

arXiv: https://arxiv.org/abs/2502.11368 · HTML: https://mlsanity.com/paper/2502.11368/

## Key facts

- Cited by 10 papers in the map (data through 7 Oct 2026).
- Builds on 14 of its 57 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 14 of 14)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks](https://mlsanity.com/paper/2305.10160/) (2023)
- [Exploring LLM Prompting Strategies for Joint Essay Scoring and Feedback Generation](https://mlsanity.com/paper/2404.15845/) (2024)
- [Automatic Text Scoring Using Neural Networks](https://mlsanity.com/paper/1606.04289/) (2016)
- [Language models and Automated Essay Scoring](https://mlsanity.com/paper/1909.09482/) (2019)
- [Finding Blind Spots in Evaluator LLMs with Interpretable Checklists](https://mlsanity.com/paper/2406.13439/) (2024)
- [LLM-as-a-tutor in EFL Writing Education: Focusing on Evaluation of Student-LLM Interaction](https://mlsanity.com/paper/2310.05191/) (2023)
- [Can GPT-4 do L2 analytic assessment?](https://mlsanity.com/paper/2404.18557/) (2024)
- [GEE! Grammar Error Explanation with Large Language Models](https://mlsanity.com/paper/2311.09517/) (2023)
- [Error syntax aware augmentation of feedback comment generation dataset](https://mlsanity.com/paper/2212.14293/) (2022)

## Built on by (most cited first, 10 of 10)

- [Opportunities and Challenges of LLMs in Education: An NLP Perspective](https://mlsanity.com/paper/2507.22753/) (2025)
- [Foundation models may exhibit staged progression in novel CBRN threat disclosure](https://mlsanity.com/paper/2503.15182/) (2025)
- [Generative AI performance in core undergraduate mathematics: a curriculum-level case study](https://mlsanity.com/paper/2509.13359/) (2025)
- ["You Are Rejected!": An Empirical Study of Large Language Models Taking Hiring Evaluations](https://mlsanity.com/paper/2510.19167/) (2025)
- [Zero-Shot Grammar Competency Estimation Using Large Language Model Generated Pseudo Labels](https://mlsanity.com/paper/2511.13152/) (2025)
- [From Prompting to Preference Optimization: A Comparative Study of LLM-based Automated Essay Scoring](https://mlsanity.com/paper/2603.06424/) (2026)
- [Towards Self-Referential Analytic Assessment: A Profile-Based Approach to L2 Writing Evaluation with LLMs](https://mlsanity.com/paper/2605.04298/) (2026)
- [FOXGLOVE: Understanding Goal-Oriented and Anchored Writing Feedback from Experts and LLMs on Argumentative Essays](https://mlsanity.com/paper/2606.06271/) (2026)
- [When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs](https://mlsanity.com/paper/2608.23978/) (2026)
- [Evaluating Feedback Focus and Pedagogical Adaptivity in LLM-Generated Feedback on Student Writing](https://mlsanity.com/paper/2609.28026/) (2026)

## Abstract (arXiv)

The paper explores the performance of LLMs in the context of multi-dimensional analytic writing assessments, i.e. their ability to provide both scores and comments based on multiple assessment criteria. Using a corpus of literature reviews written by L2 graduate students and assessed by human experts against 9 analytic criteria, we prompt several popular LLMs to perform the same task under various conditions. To evaluate the quality of feedback comments, we apply a novel feedback comment quality evaluation framework. This framework is interpretable, cost-efficient, scalable, and reproducible, compared to existing methods that rely on manual judgments. We find that LLMs can generate reasonably good and generally reliable multi-dimensional analytic assessments. We release our corpus and code for reproducibility.