# Finetuning Transformer Models to Build ASAG System

arXiv: https://arxiv.org/abs/2109.12300 · HTML: https://mlsanity.com/paper/2109.12300/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 81 references that are in the map.
- First posted on arXiv 25 Sep 2021.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Using Word Embeddings for Automatic Query Expansion](https://mlsanity.com/paper/1606.07608/) (2016)
- [Automatic Short Answer Grading and Feedback Using Text Mining Methods](https://mlsanity.com/paper/1807.10543/) (2018)
- [Automatic Short Answer Grading via Multiway Attention Networks](https://mlsanity.com/paper/1909.10166/) (2019)

## Built on by (most cited first, 1 of 1)

- [Concept-based Rubrics Improve LLM Formative Assessment and Data Synthesis](https://mlsanity.com/paper/2504.03877/) (2025)

## Abstract (arXiv)

Research towards creating systems for automatic grading of student answers to quiz and exam questions in educational settings has been ongoing since 1966. Over the years, the problem was divided into many categories. Among them, grading text answers were divided into short answer grading, and essay grading. The goal of this work was to develop an ML-based short answer grading system. I hence built a system which uses finetuning on Roberta Large Model pretrained on STS benchmark dataset and have also created an interface to show the production readiness of the system. I evaluated the performance of the system on the Mohler extended dataset and SciEntsBank Dataset. The developed system achieved a Pearsons Correlation of 0.82 and RMSE of 0.7 on the Mohler Dataset which beats the SOTA performance on this dataset which is correlation of 0.805 and RMSE of 0.793. Additionally, Pearsons Correlation of 0.79 and RMSE of 0.56 was achieved on the SciEntsBank Dataset, which only reconfirms the robustness of the system. A few observations during achieving these results included usage of batch size of 1 produced better results than using batch size of 16 or 32 and using huber loss as loss function performed well on this regression task. The system was tried and tested on train and validation splits using various random seeds and still has been tweaked to achieve a minimum of 0.76 of correlation and a maximum 0.15 (out of 1) RMSE on any dataset.