# Can LLMs Identify Gaps and Misconceptions in Students' Code Explanations?

arXiv: https://arxiv.org/abs/2501.10365 · HTML: https://mlsanity.com/paper/2501.10365/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- First posted on arXiv 9 Dec 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Built on by (most cited first, 4 of 4)

- [AI In Cybersecurity Education -- Scalable Agentic CTF Design Principles and Educational Outcomes](https://mlsanity.com/paper/2603.21551/) (2026)
- [Exploring the Effectiveness of Using LLMs for Automated Assessment of Student Self Explanations in Programming Education](https://mlsanity.com/paper/2605.21614/) (2026)
- [Taxonomy-Targeted Error Generation for Quantitative Reasoning](https://mlsanity.com/paper/2605.29007/) (2026)
- [Self-Explanation Tutor for Active Study of CS1 Worked Examples](https://mlsanity.com/paper/2608.25180/) (2026)

## Abstract (arXiv)

This paper investigates various approaches using Large Language Models (LLMs) to identify gaps and misconceptions in students' self-explanations of specific instructional material, in our case explanations of code examples. This research is a part of our larger effort to automate the assessment of students' freely generated responses, focusing specifically on their self-explanations of code examples during activities related to code comprehension. In this work, we experiment with zero-shot prompting, Supervised Fine-Tuning (SFT), and preference alignment of LLMs to identify gaps in students' self-explanation. With simple prompting, GPT-4 consistently outperformed LLaMA3 and Mistral in identifying gaps and misconceptions, as confirmed by human evaluations. Additionally, our results suggest that fine-tuned large language models are more effective at identifying gaps in students' explanations compared to zero-shot and few-shot prompting techniques. Furthermore, our findings show that the preference optimization approach using Odds Ratio Preference Optimization (ORPO) outperforms SFT in identifying gaps and misconceptions in students' code explanations.