mlsanityData to 13 Oct 2026

Automated Refinement of Essay Scoring Rubrics for Language Models via Reflect-and-Revise

arXiv 2510.09030 ↗First posted 10 Oct 2025cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

The performance of Large Language Models (LLMs) is highly sensitive to the prompts they are given. Drawing inspiration from the field of prompt optimization, this study investigates the potential for enhancing Automated Essay Scoring (AES) by refining the scoring rubrics used by LLMs. Specifically, our approach prompts models to iteratively refine rubrics by reflecting on models' own scoring rationales and observed discrepancies with human scores on sample essays. Experiments on the TOEFL11 and ASAP datasets using GPT-4.1, Gemini-2.5-Pro, and Qwen-3-Next-80B-A3B-Instruct show Quadratic Weighted Kappa (QWK) improvements of up to 0.19 and 0.47, respectively. Notably, even with a simple initial rubric, our approach achieves comparable or better QWK than using detailed human-authored rubrics. Our findings highlight the importance of iterative rubric refinement in LLM-based AES to enhance alignment with human evaluations.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Builds on 20 of its 43 references that are in the map. First posted on arXiv 10 Oct 2025.

Builds on

20 of 43 references.

References · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,856
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,367
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,268
  4. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 814
  5. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 489
  6. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 345
  7. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 274
  8. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 233
  9. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 212
  10. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 132
  11. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 116
  12. 2025Reinforcement Learning with Rubric Anchorscited by 110
8 more
  1. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 91
  2. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decompositioncited by 33
  3. 2024Unleashing Large Language Models' Proficiency in Zero-shot Essay Scoringcited by 33
  4. 2024LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarioscited by 32
  5. 2025Are Checklists Really Useful for Automatic Evaluation of Generative Tasks?cited by 9
  6. 2024Large Language Models Are Active Critics in NLG Evaluationcited by 6
  7. 2025HPSS: Heuristic Prompting Strategy Search for LLM Evaluatorscited by 5
  8. 2025Do We Need a Detailed Rubric for Automated Essay Scoring using Large Language Models?cited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  2. 2026Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Qualitycited by 4
  3. 2026A Unified Framework to Elicit Structured Feedback for Interpretable Multi-Trait Essay Scoringcited by 0