# FeedEval: Pedagogically Aligned Evaluation of LLM-Generated Essay Feedback

arXiv: https://arxiv.org/abs/2601.04574 · HTML: https://mlsanity.com/paper/2601.04574/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 23 of its 54 references that are in the map.
- First posted on arXiv 8 Jan 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 23 of 23)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks](https://mlsanity.com/paper/2305.18395/) (2023)
- [Exploring LLM Prompting Strategies for Joint Essay Scoring and Feedback Generation](https://mlsanity.com/paper/2404.15845/) (2024)
- [Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors](https://mlsanity.com/paper/2412.09416/) (2024)
- [MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors](https://mlsanity.com/paper/2502.18940/) (2025)
- [On the Use of BERT for Automated Essay Scoring: Joint Learning of Multi-Scale Essay Representation](https://mlsanity.com/paper/2205.03835/) (2022)
- [MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactions](https://mlsanity.com/paper/2405.19444/) (2024)
- [Prompt- and Trait Relation-aware Cross-prompt Essay Trait Scoring](https://mlsanity.com/paper/2305.16826/) (2023)
- [Learning to Summarize from LLM-generated Feedback](https://mlsanity.com/paper/2410.13116/) (2024)
- [Rationale Behind Essay Scores: Enhancing S-LLM's Multi-Trait Essay Scoring with Rationale Generated by LLMs](https://mlsanity.com/paper/2410.14202/) (2024)
- [Improving the Validity of Automatically Generated Feedback via Reinforcement Learning](https://mlsanity.com/paper/2403.01304/) (2024)
- [Verifiable Format Control for Large Language Model Generations](https://mlsanity.com/paper/2502.04498/) (2025)
- [Distilling ChatGPT for Explainable Automated Student Answer Assessment](https://mlsanity.com/paper/2305.12962/) (2023)
- [Distilling Reasoning Ability from Large Language Models with Adaptive Thinking](https://mlsanity.com/paper/2404.09170/) (2024)
- [Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluation](https://mlsanity.com/paper/2409.07355/) (2024)
- [Autoregressive Score Generation for Multi-trait Essay Scoring](https://mlsanity.com/paper/2403.08332/) (2024)
- [RECIPE4U: Student-ChatGPT Interaction Dataset in EFL Writing Education](https://mlsanity.com/paper/2403.08272/) (2024)
- [Teach-to-Reason with Scoring: Self-Explainable Rationale-Driven Multi-Trait Essay Scoring](https://mlsanity.com/paper/2502.20748/) (2025)
- [FEAT: A Preference Feedback Dataset through a Cost-Effective Auto-Generation and Labeling Framework for English AI Tutoring](https://mlsanity.com/paper/2506.19325/) (2025)

## Built on by (most cited first, 4 of 4)

- [Investigating Learner-Aware Design of LLM-Generated Educational Feedback](https://mlsanity.com/paper/2602.11650/) (2026)
- [Generating Constructive Feedback on Stories via Reinforcement Learning](https://mlsanity.com/paper/2609.04824/) (2026)
- [Evaluating Feedback Focus and Pedagogical Adaptivity in LLM-Generated Feedback on Student Writing](https://mlsanity.com/paper/2609.28026/) (2026)
- [How Well Can LLMs Simulate Real Learner Evaluations of Educational Feedback?](https://mlsanity.com/paper/2609.35376/) (2026)

## Abstract (arXiv)

Going beyond the prediction of numerical scores, recent research in automated essay scoring has increasingly emphasized the generation of high-quality feedback that provides justification and actionable guidance. To mitigate the high cost of expert annotation, prior work has commonly relied on LLM-generated feedback to train essay assessment models. However, such feedback is often incorporated without explicit quality validation, resulting in the propagation of noise in downstream applications. To address this limitation, we propose FeedEval, an LLM-based framework for evaluating LLM-generated essay feedback along three pedagogically grounded dimensions: specificity, helpfulness, and validity. FeedEval employs dimension-specialized LLM evaluators trained on datasets curated in this study to assess multiple feedback candidates and select high-quality feedback for downstream use. Experiments on the ASAP++ benchmark show that FeedEval closely aligns with human expert judgments and that essay scoring models trained with FeedEval-filtered high-quality feedback achieve superior scoring performance. Furthermore, revision experiments using small LLMs show that the high-quality feedback identified by FeedEval leads to more effective essay revisions. We release our code and curated datasets at: https://github.com/BBeeChu/FeedEval.git.