# Towards Trustworthy AutoGrading of Short, Multi-lingual, Multi-type Answers

arXiv: https://arxiv.org/abs/2201.03425 · HTML: https://mlsanity.com/paper/2201.03425/

## Key facts

- Cited by 6 papers in the map (data through 13 Oct 2026).
- Builds on 15 of its 59 references that are in the map.
- First posted on arXiv 2 Jan 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/1801.00553/) (2018)
- [A comparative study of fairness-enhancing interventions in machine learning](https://mlsanity.com/paper/1802.04422/) (2018)
- [Deep Learning is Robust to Massive Label Noise](https://mlsanity.com/paper/1705.10694/) (2017)
- [Deep Learning on Small Datasets without Pre-Training using Cosine Loss](https://mlsanity.com/paper/1901.09054/) (2019)
- [A Comparison of LSTM and BERT for Small Corpus](https://mlsanity.com/paper/2009.05451/) (2020)
- [Personalization of Deep Learning](https://mlsanity.com/paper/1909.02803/) (2019)
- [Automatic Short Answer Grading and Feedback Using Text Mining Methods](https://mlsanity.com/paper/1807.10543/) (2018)
- [Human-to-AI Coach: Improving Human Inputs to AI Systems](https://mlsanity.com/paper/1912.03652/) (2019)

## Built on by (most cited first, 6 of 6)

- [Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review](https://mlsanity.com/paper/2303.13379/) (2023)
- [Unleashing the potential of prompt engineering for large language models](https://mlsanity.com/paper/2310.14735/) (2023)
- [Towards LLM-based Autograding for Short Textual Answers](https://mlsanity.com/paper/2309.11508/) (2023)
- [Foundation models in brief: A historical, socio-technical focus](https://mlsanity.com/paper/2212.08967/) (2022)
- [Enhancing Security and Strengthening Defenses in Automated Short-Answer Grading Systems](https://mlsanity.com/paper/2505.00061/) (2025)
- [Autograder+: A Multi-Faceted AI Framework for Rich Pedagogical Feedback in Programming Education](https://mlsanity.com/paper/2510.26402/) (2025)

## Abstract (arXiv)

Autograding short textual answers has become much more feasible due to the rise of NLP and the increased availability of question-answer pairs brought about by a shift to online education. Autograding performance is still inferior to human grading. The statistical and black-box nature of state-of-the-art machine learning models makes them untrustworthy, raising ethical concerns and limiting their practical utility. Furthermore, the evaluation of autograding is typically confined to small, monolingual datasets for a specific question type. This study uses a large dataset consisting of about 10 million question-answer pairs from multiple languages covering diverse fields such as math and language, and strong variation in question and answer syntax. We demonstrate the effectiveness of fine-tuning transformer models for autograding for such complex datasets. Our best hyperparameter-tuned model yields an accuracy of about 86.5%, comparable to the state-of-the-art models that are less general and more tuned to a specific type of question, subject, and language. More importantly, we address trust and ethical concerns. By involving humans in the autograding process, we show how to improve the accuracy of automatically graded answers, achieving accuracy equivalent to that of teaching assistants. We also show how teachers can effectively control the type of errors made by the system and how they can validate efficiently that the autograder's performance on individual exams is close to the expected performance.