# Analysis of Student-LLM Interaction in a Software Engineering Project

arXiv: https://arxiv.org/abs/2502.01273 · HTML: https://mlsanity.com/paper/2502.01273/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 23 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Builds on (most cited first, 9 of 9)

- [CodePlan: Repository-level Coding using LLMs and Planning](https://mlsanity.com/paper/2309.12499/) (2023)
- [Few-shot training LLMs for project-specific code-summarization](https://mlsanity.com/paper/2207.04237/) (2022)
- [InferFix: End-to-End Program Repair with LLMs](https://mlsanity.com/paper/2303.07263/) (2023)
- [ChatUniTest: A Framework for LLM-Based Test Generation](https://mlsanity.com/paper/2305.04764/) (2023)
- [CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs](https://mlsanity.com/paper/2401.11314/) (2024)
- [LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2308.11148/) (2023)
- [Empowering Personalized Learning through a Conversation-based Tutoring System with Student Modeling](https://mlsanity.com/paper/2403.14071/) (2024)
- [An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project](https://mlsanity.com/paper/2401.16186/) (2024)
- [Efficient Classification of Student Help Requests in Programming Courses Using Large Language Models](https://mlsanity.com/paper/2310.20105/) (2023)

## Built on by (most cited first, 1 of 1)

- [Conversational AI for Rapid Scientific Prototyping: A Case Study on ESA's ELOPE Competition](https://mlsanity.com/paper/2601.04920/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are becoming increasingly competent across various domains, educators are showing a growing interest in integrating these LLMs into the learning process. Especially in software engineering, LLMs have demonstrated qualitatively better capabilities in code summarization, code generation, and debugging. Despite various research on LLMs for software engineering tasks in practice, limited research captures the benefits of LLMs for pedagogical advancements and their impact on the student learning process. To this extent, we analyze 126 undergraduate students' interaction with an AI assistant during a 13-week semester to understand the benefits of AI for software engineering learning. We analyze the conversations, code generated, code utilized, and the human intervention levels to integrate the code into the code base. Our findings suggest that students prefer ChatGPT over CoPilot. Our analysis also finds that ChatGPT generates responses with lower computational complexity compared to CoPilot. Furthermore, conversational-based interaction helps improve the quality of the code generated compared to auto-generated code. Early adoption of LLMs in software engineering is crucial to remain competitive in the rapidly developing landscape. Hence, the next generation of software engineers must acquire the necessary skills to interact with AI to improve productivity.