# Supervised Fine-Tuning LLMs to Behave as Pedagogical Agents in Programming Education

arXiv: https://arxiv.org/abs/2502.20527 · HTML: https://mlsanity.com/paper/2502.20527/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Educational agents and math problem solving, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Placed in

- LLM agents and tool use / [Educational agents and math problem solving](https://mlsanity.com/agents/educational-agents-and-math-problem-solving/)

## Builds on (most cited first, 20 of 20)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?](https://mlsanity.com/paper/2405.05904/) (2024)
- [Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment](https://mlsanity.com/paper/2312.12148/) (2023)
- [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022)
- [Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language](https://mlsanity.com/paper/2210.15157/) (2022)
- [CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes](https://mlsanity.com/paper/2308.06921/) (2023)
- [The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities](https://mlsanity.com/paper/2408.13296/) (2024)
- [Computing Education in the Era of Generative AI](https://mlsanity.com/paper/2306.02608/) (2023)
- [The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers](https://mlsanity.com/paper/2405.17739/) (2024)
- [DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents](https://mlsanity.com/paper/2311.09805/) (2023)
- [Fine-Tuning and Prompt Engineering for Large Language Models-based Code Review Automation](https://mlsanity.com/paper/2402.00905/) (2024)
- [Desirable Characteristics for AI Teaching Assistants in Programming Education](https://mlsanity.com/paper/2405.14178/) (2024)
- [Lifelong Personalized Low-Rank Adaptation of Large Language Models for Recommendation](https://mlsanity.com/paper/2408.03533/) (2024)
- [A review on the use of large language models as virtual tutors](https://mlsanity.com/paper/2405.11983/) (2024)
- [FineTuneBench: How well do commercial fine-tuning APIs infuse knowledge into LLMs?](https://mlsanity.com/paper/2411.05059/) (2024)
- [Dcc --help: Generating Context-Aware Compiler Error Explanations with Large Language Models](https://mlsanity.com/paper/2308.11873/) (2023)
- [From Solitary Directives to Interactive Encouragement! LLM Secure Code Generation by Natural Language Prompting](https://mlsanity.com/paper/2410.14321/) (2024)

## Built on by (most cited first, 4 of 4)

- [Narrowing the Gap: Supervised Fine-Tuning of Open-Source LLMs as a Viable Alternative to Proprietary Models for Pedagogical Tools](https://mlsanity.com/paper/2507.05305/) (2025)
- [How Does Prefix Matter in Reasoning Model Tuning?](https://mlsanity.com/paper/2601.01624/) (2026)
- [What Hard Tokens Reveal: Exploiting Low-confidence Tokens for Membership Inference Attacks against Large Language Models](https://mlsanity.com/paper/2601.20885/) (2026)
- [SocraticRepair: Planning-Guided Socratic Tutoring for Program Debugging and Misconception Repair](https://mlsanity.com/paper/2610.15204/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly being explored in higher education, yet their effectiveness as teaching agents remains underexamined. In this paper, we present the development of GuideLM, a fine-tuned LLM designed for programming education. GuideLM has been integrated into the Debugging C Compiler (DCC), an educational C compiler that leverages LLMs to generate pedagogically sound error explanations. Previously, DCC relied on off-the-shelf OpenAI models, which, while accurate, often over-assisted students by directly providing solutions despite contrary prompting. To address this, we employed supervised fine-tuning (SFT) on a dataset of 528 student-question/teacher-answer pairs, creating two models: GuideLM and GuideLM-mini, fine-tuned on ChatGPT-4o and 4o-mini, respectively. We conducted an expert analysis of 400 responses per model, comparing their pedagogical effectiveness against base OpenAI models. Our evaluation, grounded in constructivism and cognitive load theory, assessed factors such as conceptual scaffolding, clarity, and Socratic guidance. Results indicate that GuideLM and GuideLM-mini improve pedagogical performance, with an 8% increase in Socratic guidance and a 58% improvement in economy of words compared to GPT-4o. However, this refinement comes at the cost of a slight reduction in general accuracy. While further work is needed, our findings suggest that fine-tuning LLMs with targeted datasets is a promising approach for developing models better suited to educational contexts.