# Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study

arXiv: https://arxiv.org/abs/2404.13414 · HTML: https://mlsanity.com/paper/2404.13414/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2024.

## Built on by (most cited first, 26 of 26)

- [Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debugging](https://mlsanity.com/paper/2406.11709/) (2024)
- [Large Language Models in Computer Science Education: A Systematic Literature Review](https://mlsanity.com/paper/2410.16349/) (2024)
- [TutorBench: A Benchmark To Assess Tutoring Capabilities Of Large Language Models](https://mlsanity.com/paper/2510.02663/) (2025)
- [Faculty Perspectives on the Potential of RAG in Computer Science Higher Education](https://mlsanity.com/paper/2408.01462/) (2024)
- [Automated Assignment Grading with Large Language Models: Insights From a Bioinformatics Course](https://mlsanity.com/paper/2501.14499/) (2025)
- [From Idea to Implementation: Evaluating the Influence of Large Language Models in Software Development -- An Opinion Paper](https://mlsanity.com/paper/2503.07450/) (2025)
- [Towards responsible AI for education: Hybrid human-AI to confront the Elephant in the room](https://mlsanity.com/paper/2504.16148/) (2025)
- [Deliberate Lab: A Platform for Real-Time Human-AI Social Experiments](https://mlsanity.com/paper/2510.13011/) (2025)
- [The Lazy Student's Dream: ChatGPT Passing an Engineering Course on Its Own](https://mlsanity.com/paper/2503.05760/) (2025)
- [A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics](https://mlsanity.com/paper/2505.15469/) (2025)
- [Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education](https://mlsanity.com/paper/2505.18220/) (2025)
- [Problems With Large Language Models for Learner Modelling: Why LLMs Alone Fall Short for Responsible Tutoring in K--12 Education](https://mlsanity.com/paper/2512.23036/) (2025)
- [A Knowledge-Component-Based Methodology for Evaluating AI Assistants](https://mlsanity.com/paper/2406.05603/) (2024)
- [Utilizing ChatGPT in a Data Structures and Algorithms Course: A Teaching Assistant's Perspective](https://mlsanity.com/paper/2410.08899/) (2024)
- [Arti-"fickle" Intelligence: Using LLMs as a Tool for Inference in the Political and Social Sciences](https://mlsanity.com/paper/2504.03822/) (2025)
- [JELAI: Integrating AI and Learning Analytics in Jupyter Notebooks](https://mlsanity.com/paper/2505.17593/) (2025)
- [Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?](https://mlsanity.com/paper/2507.17258/) (2025)
- [Assessing LLM Response Quality in the Context of Technology-Facilitated Abuse](https://mlsanity.com/paper/2602.17672/) (2026)
- [CODE-GEN: A Human-in-the-Loop RAG-Based Agentic AI System for Multiple-Choice Question Generation](https://mlsanity.com/paper/2604.03926/) (2026)
- [Neural-Symbolic Knowledge Tracing: Injecting Educational Knowledge into Deep Learning for Responsible Learner Modelling](https://mlsanity.com/paper/2604.08263/) (2026)
- [Objective Metrics for Evaluating Large Language Models Using External Data Sources](https://mlsanity.com/paper/2508.08277/) (2025)
- [DebugTA: An LLM-Based Agent for Simplifying Debugging and Teaching in Programming Education](https://mlsanity.com/paper/2510.11076/) (2025)
- [Hybrid LLM-Embedded Dialogue Agents for Learner Reflection: Designing Responsive and Theory-Driven Interactions](https://mlsanity.com/paper/2602.20486/) (2026)
- [Kwame 2.0: Human-in-the-Loop Generative AI Teaching Assistant for Large Scale Online Coding Education in Africa](https://mlsanity.com/paper/2603.29159/) (2026)
- [Evaluating Large Language Models on Computer Science University Exams in Data Structures](https://mlsanity.com/paper/2604.23347/) (2026)
- [The GenAI Skill Bypass: Mapping Divergent Pathways of University Students and Staff AI Literacy](https://mlsanity.com/paper/2607.05411/) (2026)

## Abstract (arXiv)

The integration of AI assistants, especially through the development of Large Language Models (LLMs), into computer science education has sparked significant debate. An emerging body of work has looked into using LLMs in education, but few have examined the impacts of LLMs on students in entry-level programming courses, particularly in real-world contexts and over extended periods. To address this research gap, we conducted a semester-long, between-subjects study with 50 students using CodeTutor, an LLM-powered assistant developed by our research team. Our study results show that students who used CodeTutor (the experimental group) achieved statistically significant improvements in their final scores compared to peers who did not use the tool (the control group). Within the experimental group, those without prior experience with LLM-powered tools demonstrated significantly greater performance gain than their counterparts. We also found that students expressed positive feedback regarding CodeTutor's capability, though they also had concerns about CodeTutor's limited role in developing critical thinking skills. Over the semester, students' agreement with CodeTutor's suggestions decreased, with a growing preference for support from traditional human teaching assistants. Our analysis further reveals that the quality of user prompts was significantly correlated with CodeTutor's response effectiveness. Building upon our results, we discuss the implications of our findings for integrating Generative AI literacy into curricula to foster critical thinking skills and turn to examining the temporal dynamics of user engagement with LLM-powered tools. We further discuss the discrepancy between the anticipated functions of tools and students' actual capabilities, which sheds light on the need for tailored strategies to improve educational outcomes.