# Fine-tuning ChatGPT for Automatic Scoring

arXiv: https://arxiv.org/abs/2310.10072 · HTML: https://mlsanity.com/paper/2310.10072/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 80 references that are in the map.
- First posted on arXiv 16 Oct 2023.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [Is ChatGPT a Good Recommender? A Preliminary Study](https://mlsanity.com/paper/2304.10149/) (2023)
- [Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review](https://mlsanity.com/paper/2303.13379/) (2023)
- [MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets](https://mlsanity.com/paper/2308.12067/) (2023)
- [MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education](https://mlsanity.com/paper/2106.07340/) (2021)
- [Language models and Automated Essay Scoring](https://mlsanity.com/paper/1909.09482/) (2019)
- [Context Matters: A Strategy to Pre-train Language Model for Science Education](https://mlsanity.com/paper/2301.12031/) (2023)
- [On the application of Large Language Models for language teaching and assessment technology](https://mlsanity.com/paper/2307.08393/) (2023)
- [Matching Exemplar as Next Sentence Prediction (MeNSP): Zero-shot Prompt Learning for Automatic Scoring in Science Education](https://mlsanity.com/paper/2301.08771/) (2023)
- [Assessing Student Errors in Experimentation Using Artificial Intelligence and Large Language Models: A Comparative Study with Human Raters](https://mlsanity.com/paper/2308.06088/) (2023)
- [Automated essay scoring using efficient transformer-based language models](https://mlsanity.com/paper/2102.13136/) (2021)
- [Pseudo AI Bias](https://mlsanity.com/paper/2210.08141/) (2022)
- [h2oGPT: Democratizing Large Language Models](https://mlsanity.com/paper/2306.08161/) (2023)
- [Automatic Scoring of Dream Reports' Emotional Content with Large Language Models](https://mlsanity.com/paper/2302.14828/) (2023)

## Built on by (most cited first, 30 of 51)

- [Applying Large Language Models and Chain-of-Thought for Automatic Scoring](https://mlsanity.com/paper/2312.03748/) (2023)
- [Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era](https://mlsanity.com/paper/2403.08946/) (2024)
- [Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning](https://mlsanity.com/paper/2509.24372/) (2025)
- [Multimodality of AI for Education: Towards Artificial General Intelligence](https://mlsanity.com/paper/2312.06037/) (2023)
- [Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)](https://mlsanity.com/paper/2505.21091/) (2025)
- [A LLM-Powered Automatic Grading Framework with Human-Level Guidelines Optimization](https://mlsanity.com/paper/2410.02165/) (2024)
- [Taking the Next Step with Generative Artificial Intelligence: The Transformative Role of Multimodal Large Language Models in Science Education](https://mlsanity.com/paper/2401.00832/) (2024)
- [NERIF: GPT-4V for Automatic Scoring of Drawn Models](https://mlsanity.com/paper/2311.12990/) (2023)
- [Unveiling Scoring Processes: Dissecting the Differences between LLMs and Human Graders in Automatic Scoring](https://mlsanity.com/paper/2407.18328/) (2024)
- [Beyond IID: Optimizing Instruction Learning from the Perspective of Instruction Interaction and Dependency](https://mlsanity.com/paper/2409.07045/) (2024)
- [AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognition](https://mlsanity.com/paper/2509.21910/) (2025)
- [FineTuneBench: How well do commercial fine-tuning APIs infuse knowledge into LLMs?](https://mlsanity.com/paper/2411.05059/) (2024)
- [How Can I Improve? Using GPT to Highlight the Desired and Undesired Parts of Open-ended Responses](https://mlsanity.com/paper/2405.00291/) (2024)
- [Harnessing Large Language Models Over Transformer Models for Detecting Bengali Depressive Social Media Text: A Comprehensive Study](https://mlsanity.com/paper/2401.07310/) (2024)
- [How well can LLMs Grade Essays in Arabic?](https://mlsanity.com/paper/2501.16516/) (2025)
- [Towards Adaptive Feedback with AI: Comparing the Feedback Quality of LLMs and Teachers on Experimentation Protocols](https://mlsanity.com/paper/2502.12842/) (2025)
- [Detecting Visual Information Manipulation Attacks in Augmented Reality: A Multimodal Semantic Reasoning Approach](https://mlsanity.com/paper/2507.20356/) (2025)
- [Knowledge Distillation of LLM for Automatic Scoring of Science Education Assessments](https://mlsanity.com/paper/2312.15842/) (2023)
- [Generating Planning Feedback for Open-Ended Programming Exercises with LLMs](https://mlsanity.com/paper/2504.08958/) (2025)
- [Exploring LLM Autoscoring Reliability in Large-Scale Writing Assessments Using Generalizability Theory](https://mlsanity.com/paper/2507.19980/) (2025)
- [Automatic Scoring of Students' Science Writing Using Hybrid Neural Network](https://mlsanity.com/paper/2312.03752/) (2023)
- [AI and Machine Learning for Next Generation Science Assessments](https://mlsanity.com/paper/2405.06660/) (2024)
- [Automatic Essay Multi-dimensional Scoring with Fine-tuning and Multiple Regression](https://mlsanity.com/paper/2406.01198/) (2024)
- [The Life Cycle of Large Language Models: A Review of Biases in Education](https://mlsanity.com/paper/2407.11203/) (2024)
- [Fine-tuning ChatGPT for Automatic Scoring of Written Scientific Explanations in Chinese](https://mlsanity.com/paper/2501.06704/) (2025)
- [Towards responsible AI for education: Hybrid human-AI to confront the Elephant in the room](https://mlsanity.com/paper/2504.16148/) (2025)
- [Generalizable and Efficient Automated Scoring with a Knowledge-Distilled Multi-Task Mixture-of-Experts](https://mlsanity.com/paper/2511.17601/) (2025)
- [G-SciEdBERT: A Contextualized LLM for Science Assessment Tasks in German](https://mlsanity.com/paper/2402.06584/) (2024)
- [Using ChatGPT to Score Essays and Short-Form Constructed Responses](https://mlsanity.com/paper/2408.09540/) (2024)
- [Zero-Shot Classification of Crisis Tweets Using Instruction-Finetuned Large Language Models](https://mlsanity.com/paper/2410.00182/) (2024)

## Abstract (arXiv)

This study highlights the potential of fine-tuned ChatGPT (GPT-3.5) for automatically scoring student written constructed responses using example assessment tasks in science education. Recent studies on OpenAI's generative model GPT-3.5 proved its superiority in predicting the natural language with high accuracy and human-like responses. GPT-3.5 has been trained over enormous online language materials such as journals and Wikipedia; therefore, more than direct usage of pre-trained GPT-3.5 is required for automatic scoring as students utilize a different language than trained material. These imply that a domain-specific model, fine-tuned over data for specific tasks, can enhance model performance. In this study, we fine-tuned GPT-3.5 on six assessment tasks with a diverse dataset of middle-school and high-school student responses and expert scoring. The six tasks comprise two multi-label and four multi-class assessment tasks. We compare the performance of fine-tuned GPT-3.5 with the fine-tuned state-of-the-art Google's generated language model, BERT. The results show that in-domain training corpora constructed from science questions and responses for BERT achieved average accuracy = 0.838, SD = 0.069. GPT-3.5 shows a remarkable average increase (9.1%) in automatic scoring accuracy (mean = 9.15, SD = 0.042) for the six tasks, p =0.001 < 0.05. Specifically, for multi-label tasks (item 1 with 5 labels; item 2 with 10 labels), GPT-3.5 achieved significantly higher scoring accuracy than BERT across all the labels, with the second item achieving a 7.1% increase. The average scoring increase for the four multi-class items for GPT-3.5 was 10.6% compared to BERT. Our study confirmed the effectiveness of fine-tuned GPT-3.5 for automatic scoring of student responses on domain-specific data in education with high accuracy. We have released fine-tuned models for public use and community engagement.