# Exploring the Effectiveness of Using LLMs for Automated Assessment of Student Self Explanations in Programming Education

arXiv: https://arxiv.org/abs/2605.21614 · HTML: https://mlsanity.com/paper/2605.21614/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 20 May 2026.

## Builds on (most cited first, 14 of 14)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Applying Large Language Models and Chain-of-Thought for Automatic Scoring](https://mlsanity.com/paper/2312.03748/) (2023)
- [RevisEval: Improving LLM-as-a-Judge via Response-Adapted References](https://mlsanity.com/paper/2410.05193/) (2024)
- [GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizers](https://mlsanity.com/paper/2412.09722/) (2024)
- [Explaining Code with a Purpose: An Integrated Approach for Developing Code Comprehension and Prompting Skills](https://mlsanity.com/paper/2403.06050/) (2024)
- [eRevise+RF: A Writing Evaluation System for Assessing Student Essay Revisions and Providing Formative Feedback](https://mlsanity.com/paper/2501.00715/) (2025)
- [Improve LLM-based Automatic Essay Scoring with Linguistic Features](https://mlsanity.com/paper/2502.09497/) (2025)
- [Predicting the Quality of Revisions in Argumentative Writing](https://mlsanity.com/paper/2306.00667/) (2023)
- [KASER: Knowledge-Aligned Student Error Simulator for Open-Ended Coding Tasks](https://mlsanity.com/paper/2601.06633/) (2026)
- [Cheating Automatic Short Answer Grading: On the Adversarial Usage of Adjectives and Adverbs](https://mlsanity.com/paper/2201.08318/) (2022)
- [Can LLMs Identify Gaps and Misconceptions in Students' Code Explanations?](https://mlsanity.com/paper/2501.10365/) (2024)
- [Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variation](https://mlsanity.com/paper/2505.23368/) (2025)

## Abstract (arXiv)

Worked examples are step-by-step solutions to problems in a specific domain, offered to students to acquire domain-specific problem-solving skills. The effectiveness of worked examples could be enhanced by combining them with self-explanations, which ask students to explain rather than passively study each problem-solving step. The main challenge of this approach is assessing the correctness of the student's explanations. In the prevailing approach, student explanations are judged by their semantic similarity to an instructor's or domain expert's explanation. Given recent advances in LLM-based automated scoring, it remains unclear whether semantic similarity methods are still the most effective technique to automatically score textual student responses like essays or code explanations. Comparing these methods also requires quality datasets that offer distinctive features such as balanced class distributions and domain-specific labeled data for automated scoring tasks. In this paper, we present a rigorous comparison between LLMs and semantic similarity used for automated scoring, framed as a binary classification task.