# AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognition

arXiv: https://arxiv.org/abs/2509.21910 · HTML: https://mlsanity.com/paper/2509.21910/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 53 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025)
- [Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2406.07791/) (2024)
- [What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineering](https://mlsanity.com/paper/2406.12334/) (2024)
- [AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML](https://mlsanity.com/paper/2410.02958/) (2024)
- [Applying Large Language Models and Chain-of-Thought for Automatic Scoring](https://mlsanity.com/paper/2312.03748/) (2023)
- [Fine-tuning ChatGPT for Automatic Scoring](https://mlsanity.com/paper/2310.10072/) (2023)
- [Can Large Language Models Automatically Score Proficiency of Written Essays?](https://mlsanity.com/paper/2403.06149/) (2024)
- [Unleashing Large Language Models' Proficiency in Zero-shot Essay Scoring](https://mlsanity.com/paper/2404.04941/) (2024)
- [Context Matters: A Strategy to Pre-train Language Model for Science Education](https://mlsanity.com/paper/2301.12031/) (2023)
- [Prompt Agnostic Essay Scorer: A Domain Generalization Approach to Cross-prompt Automated Essay Scoring](https://mlsanity.com/paper/2008.01441/) (2020)
- [Rationale Behind Essay Scores: Enhancing S-LLM's Multi-Trait Essay Scoring with Rationale Generated by LLMs](https://mlsanity.com/paper/2410.14202/) (2024)
- [Can AI grade your essays? A comparative analysis of large language models and teacher ratings in multidimensional essay scoring](https://mlsanity.com/paper/2411.16337/) (2024)
- [Matching Exemplar as Next Sentence Prediction (MeNSP): Zero-shot Prompt Learning for Automatic Scoring in Science Education](https://mlsanity.com/paper/2301.08771/) (2023)
- [Unveiling Scoring Processes: Dissecting the Differences between LLMs and Human Graders in Automatic Scoring](https://mlsanity.com/paper/2407.18328/) (2024)
- [Co-Attention Based Neural Network for Source-Dependent Essay Scoring](https://mlsanity.com/paper/1908.01993/) (2019)
- [Focus Directions Make Your Language Models Pay More Attention to Relevant Contexts](https://mlsanity.com/paper/2503.23306/) (2025)
- [Artificial Intelligence Bias on English Language Learners in Automatic Scoring](https://mlsanity.com/paper/2505.10643/) (2025)
- [G-SciEdBERT: A Contextualized LLM for Science Assessment Tasks in German](https://mlsanity.com/paper/2402.06584/) (2024)

## Built on by (most cited first, 13 of 13)

- [Agent-as-a-Judge](https://mlsanity.com/paper/2601.05111/) (2026)
- [From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judges](https://mlsanity.com/paper/2601.08654/) (2026)
- [World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applications](https://mlsanity.com/paper/2606.00133/) (2026)
- [A Framework for Human-AI Q-Matrix Refinement: A NeuralCDM Evaluation](https://mlsanity.com/paper/2604.16398/) (2026)
- [Generative-Evaluative Agreement: A Necessary Validity Criterion for LLM-Enabled Adaptive Assessment](https://mlsanity.com/paper/2605.19529/) (2026)
- [Specialists or Generalists? Multi-Agent and Single-Agent LLMs for Essay Grading](https://mlsanity.com/paper/2601.22386/) (2026)
- [BRIDGE the Gap: Mitigating Bias Amplification in Automated Scoring of English Language Learners via Inter-group Data Augmentation](https://mlsanity.com/paper/2602.23580/) (2026)
- [Judge Like Human Examiners: A Weighted Importance Multi-Point Evaluation Framework for Generative Tasks with Long-form Answers](https://mlsanity.com/paper/2604.11246/) (2026)
- [Learnable Assessment Skills for LLM-based Automated Scoring: Rubric Construction via Iterative Optimization](https://mlsanity.com/paper/2605.29274/) (2026)
- [When Rubrics Change: Cross-Rubric Generalization for Critical Thinking Essay Scoring](https://mlsanity.com/paper/2607.13433/) (2026)
- [Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards](https://mlsanity.com/paper/2607.19219/) (2026)
- [AgentForge: An Immersive Role-Playing Platform for Learning Agentic Software Engineering](https://mlsanity.com/paper/2608.04148/) (2026)
- [A Unified Framework to Elicit Structured Feedback for Interpretable Multi-Trait Essay Scoring](https://mlsanity.com/paper/2608.28407/) (2026)

## Abstract (arXiv)

Automated scoring plays a crucial role in education by reducing the reliance on human raters, offering scalable and immediate evaluation of student work. While large language models (LLMs) have shown strong potential in this task, their use as end-to-end raters faces challenges such as low accuracy, prompt sensitivity, limited interpretability, and rubric misalignment. These issues hinder the implementation of LLM-based automated scoring in assessment practice. To address the limitations, we propose AutoSCORE, a multi-agent LLM framework enhancing automated scoring via rubric-aligned Structured COmponent REcognition. With two agents, AutoSCORE first extracts rubric-relevant components from student responses and encodes them into a structured representation (i.e., Scoring Rubric Component Extraction Agent), which is then used to assign final scores (i.e., Scoring Agent). This design ensures that model reasoning follows a human-like grading process, enhancing interpretability and robustness. We evaluate AutoSCORE on four benchmark datasets from the ASAP benchmark, using both proprietary and open-source LLMs (GPT-4o, LLaMA-3.1-8B, and LLaMA-3.1-70B). Across diverse tasks and rubrics, AutoSCORE consistently improves scoring accuracy, human-machine agreement (QWK, correlations), and error metrics (MAE, RMSE) compared to single-agent baselines, with particularly strong benefits on complex, multi-dimensional rubrics, and especially large relative gains on smaller LLMs. These results demonstrate that structured component recognition combined with multi-agent design offers a scalable, reliable, and interpretable solution for automated scoring.