# How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering

arXiv: https://arxiv.org/abs/2312.11719 · HTML: https://mlsanity.com/paper/2312.11719/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2023.

## Built on by (most cited first, 9 of 9)

- [Human-Artificial Interaction in the Age of Agentic AI: A System-Theoretical Approach](https://mlsanity.com/paper/2502.14000/) (2025)
- [Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools](https://mlsanity.com/paper/2412.14732/) (2024)
- [The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review and Mapping Study](https://mlsanity.com/paper/2507.03156/) (2025)
- [Novice Developers' Perspectives on Adopting LLMs for Software Development: A Systematic Literature Review](https://mlsanity.com/paper/2503.07556/) (2025)
- [From Junior to Senior: Allocating Agency and Navigating Professional Growth in Agentic AI-Mediated Software Engineering](https://mlsanity.com/paper/2602.00496/) (2026)
- [Guidelines for Designing AI Technologies to Support Adult Learning](https://mlsanity.com/paper/2605.04616/) (2026)
- [Learning to Code with Context: A Study-Based Approach](https://mlsanity.com/paper/2512.05242/) (2025)
- [Agentivism: a learning theory for the age of artificial intelligence](https://mlsanity.com/paper/2604.07813/) (2026)
- [On the Prospects of Dynamic LLM Conversations in Software Development](https://mlsanity.com/paper/2608.30756/) (2026)

## Abstract (arXiv)

Conversational Generative AI (convo-genAI) is revolutionizing Software Engineering (SE) as engineers and academics embrace this technology in their work. However, there is a gap in understanding the current potential and pitfalls of this technology, specifically in supporting students in SE tasks. In this work, we evaluate through a between-subjects study (N=22) the effectiveness of ChatGPT, a convo-genAI platform, in assisting students in SE tasks. Our study did not find statistical differences in participants' productivity or self-efficacy when using ChatGPT as compared to traditional resources, but we found significantly increased frustration levels. Our study also revealed 5 distinct faults arising from violations of Human-AI interaction guidelines, which led to 7 different (negative) consequences on participants.