# Automated Text Scoring in the Age of Generative AI for the GPU-poor

arXiv: https://arxiv.org/abs/2407.01873 · HTML: https://mlsanity.com/paper/2407.01873/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 68 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment](https://mlsanity.com/paper/2312.12148/) (2023)
- [ConvBERT: Improving BERT with Span-based Dynamic Convolution](https://mlsanity.com/paper/2008.02496/) (2020)
- [Cognitive Mirage: A Review of Hallucinations in Large Language Models](https://mlsanity.com/paper/2309.06794/) (2023)
- [Exploring LLM Prompting Strategies for Joint Essay Scoring and Feedback Generation](https://mlsanity.com/paper/2404.15845/) (2024)
- [MathBERT: A Pre-Trained Model for Mathematical Formula Understanding](https://mlsanity.com/paper/2105.00377/) (2021)
- [Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning](https://mlsanity.com/paper/2311.11077/) (2023)
- [Human-AI Collaborative Essay Scoring: A Dual-Process Framework with LLMs](https://mlsanity.com/paper/2401.06431/) (2024)
- [Language models and Automated Essay Scoring](https://mlsanity.com/paper/1909.09482/) (2019)
- [Can Large Language Models Automatically Score Proficiency of Written Essays?](https://mlsanity.com/paper/2403.06149/) (2024)
- [Open, Closed, or Small Language Models for Text Classification?](https://mlsanity.com/paper/2308.10092/) (2023)
- [ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Review](https://mlsanity.com/paper/2305.03123/) (2023)
- [Get It Scored Using AutoSAS -- An Automated System for Scoring Short Answers](https://mlsanity.com/paper/2012.11243/) (2020)
- [Language Models as Science Tutors](https://mlsanity.com/paper/2402.11111/) (2024)

## Built on by (most cited first, 10 of 10)

- [Agreement Between Large Language Models and Human Raters in Essay Scoring: A Research Synthesis](https://mlsanity.com/paper/2512.14561/) (2025)
- [Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact](https://mlsanity.com/paper/2603.00883/) (2026)
- [Generative Language Models with Retrieval Augmented Generation for Automated Short Answer Scoring](https://mlsanity.com/paper/2408.03811/) (2024)
- [Cyborg Data: Merging Human with AI Generated Training Data](https://mlsanity.com/paper/2503.22736/) (2025)
- [Long Context Automated Essay Scoring with Language Models](https://mlsanity.com/paper/2509.10417/) (2025)
- [Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models](https://mlsanity.com/paper/2601.02580/) (2026)
- [Automated Essay Scoring Incorporating Annotations from Automated Feedback Systems](https://mlsanity.com/paper/2505.22771/) (2025)
- [Autoscoring Anticlimax: A Meta-analytic Understanding of AI's Short-answer Shortcomings and Wording Weaknesses](https://mlsanity.com/paper/2603.04820/) (2026)
- [Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards](https://mlsanity.com/paper/2607.19219/) (2026)
- [Multimodal Item Parameter Estimation using Simulated Response Probabilitie](https://mlsanity.com/paper/2608.10154/) (2026)

## Abstract (arXiv)

Current research on generative language models (GLMs) for automated text scoring (ATS) has focused almost exclusively on querying proprietary models via Application Programming Interfaces (APIs). Yet such practices raise issues around transparency and security, and these methods offer little in the way of efficiency or customizability. With the recent proliferation of smaller, open-source models, there is the option to explore GLMs with computers equipped with modest, consumer-grade hardware, that is, for the "GPU poor." In this study, we analyze the performance and efficiency of open-source, small-scale GLMs for ATS. Results show that GLMs can be fine-tuned to achieve adequate, though not state-of-the-art, performance. In addition to ATS, we take small steps towards analyzing models' capacity for generating feedback by prompting GLMs to explain their scores. Model-generated feedback shows promise, but requires more rigorous evaluation focused on targeted use cases.