mlsanityData to 7 Oct 2026

CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs

arXiv 2401.11314 ↗First posted 20 Jan 2024cs.HC cs.AIcited by 81

Not on the map. Why?

Abstract (arXiv)

Timely, personalized feedback is essential for students learning programming. LLM-powered tools like ChatGPT offer instant support, but reveal direct answers with code, which may hinder deep conceptual engagement. We developed CodeAid, an LLM-powered programming assistant delivering helpful, technically correct responses, without revealing code solutions. CodeAid answers conceptual questions, generates pseudo-code with line-by-line explanations, and annotates student's incorrect code with fix suggestions. We deployed CodeAid in a programming class of 700 students for a 12-week semester. A thematic analysis of 8,000 usages of CodeAid was performed, further enriched by weekly surveys, and 22 student interviews. We then interviewed eight programming educators to gain further insights. Our findings reveal four design considerations for future educational AI assistants: D1) exploiting AI's unique benefits; D2) simplifying query formulation while promoting cognitive engagement; D3) avoiding direct responses while encouraging motivated learning; and D4) maintaining transparency and control for students to asses and steer AI responses.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Builds on 16 of its 64 references that are in the map. First posted on arXiv 20 Jan 2024.

Builds on

16 of 64 references.

References · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  5. 2022Automatic Generation of Programming Exercises and Code Explanations using Large Language Modelscited by 124
  6. 2022Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generationcited by 97
  7. 2023Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programmingcited by 87
  8. 2022Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Bookcited by 84
  9. 2022Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Languagecited by 81
  10. 2023CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classescited by 71
  11. 2022Using Large Language Models to Enhance Programming Error Messagescited by 67
  12. 2023Comparing Code Explanations Created by Students and Large Language Modelscited by 63
4 more
  1. 2023"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmerscited by 62
  2. 2023How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environmentcited by 55
  3. 2023Computing Education in the Era of Generative AIcited by 50
  4. 2023Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Coursescited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 34; 2026: 21.

81 papers.

Sort:

By area:

  1. 2024The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmerscited by 46
  2. 2025PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistantscited by 44
  3. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  4. 2024Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decompositioncited by 29
  5. 2025Towards Next-Generation Recommender Systems: A Benchmark for Personalized Recommendation Assistant with LLMscited by 20
  6. 2024SirLLM: Streaming Infinite Retentive LLMcited by 16
  7. 2024Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learningcited by 16
  8. 2024Improving Socratic Question Generation using Data Augmentation and Preference Optimizationcited by 15
  9. 2024Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI Workflowcited by 13
  10. 2024Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Toolscited by 13
  11. 2025Towards User-level Private Reinforcement Learning with Human Feedbackcited by 13
  12. 2024Large Language Models in Computer Science Education: A Systematic Literature Reviewcited by 12
38 more
  1. 2024Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debuggingcited by 11
  2. 2024How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Coursecited by 11
  3. 2024Evaluating the Application of Large Language Models to Generate Feedback in Programming Educationcited by 10
  4. 2024Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Coursecited by 9
  5. 2024Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPTcited by 8
  6. 2024Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generationcited by 7
  7. 2024Leveraging Lecture Content for Improved Feedback: Explorations with GPT-4 and Retrieval Augmented Generationcited by 6
  8. 2024Understanding Communication Preferences of Information Workers in Engagement with Text-Based Conversational Agentscited by 6
  9. 2025Human-AI Experience in Integrated Development Environments: A Systematic Literature Reviewcited by 5
  10. 2025MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitivenesscited by 4
  11. 2024Pinning "Reflection" on the Agenda: Investigating Reflection in Human-LLM Co-Creation for Creative Codingcited by 3
  12. 2024ChatGPT in Data Visualization Education: A Student Perspectivecited by 3
  13. 2024One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Taskscited by 3
  14. 2024Understanding Help-Seeking Behavior of Students Using LLMs vs. Web Search for Writing SQL Queriescited by 2
  15. 2025"Would You Want an AI Tutor?" Understanding Stakeholder Perceptions of LLM-based Systems in the Classroomcited by 2
  16. 2025Improving Student-AI Interaction Through Pedagogical Prompting: An Example in Computer Science Educationcited by 2
  17. 2025ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluationcited by 2
  18. 2025Position: Thematic Analysis of Unstructured Clinical Transcripts with Large Language Modelscited by 2
  19. 2025Scaffolding Metacognition in Programming Education: Understanding Student-AI Interactions and Design Implicationscited by 2
  20. 2025Coding With AI: From a Reflection on Industrial Practices to Future Computer Science and Software Engineering Educationcited by 2
  21. 2026"Bespoke Bots": Diverse Instructor Needs for Customizing Generative AI Classroom Chatbotscited by 2
  22. 2024A Knowledge-Component-Based Methodology for Evaluating AI Assistantscited by 1
  23. 2024A GPT-based Code Review System for Programming Language Learningcited by 1
  24. 2024Integrating AI Tutors in a Programming Coursecited by 1
  25. 2024Easy Come, Easy Go? Examining the Perceptions and Learning Effects of LLM-based Chatbot in the Context of Search-as-Learningcited by 1
  26. 2025Analysis of Student-LLM Interaction in a Software Engineering Projectcited by 1
  27. 2025Prompt Programming: A Platform for Dialogue-based Computational Problem Solving with Generative AI Modelscited by 1
  28. 2025"All Roads Lead to ChatGPT": How Generative AI is Eroding Social Interactions and Student Learning Communitiescited by 1
  29. 2025JELAI: Integrating AI and Learning Analytics in Jupyter Notebookscited by 1
  30. 2025Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?cited by 1
  31. 2025Beyond Sharp Minima: Robust LLM Unlearning via Feedback-Guided Multi-Point Optimizationcited by 1
  32. 2025NeuroBridge: Using Generative AI to Bridge Cross-neurotype Communication Differences through Neurotypical Perspective-takingcited by 1
  33. 2025User Misconceptions of LLM-Based Conversational Programming Assistantscited by 1
  34. 2025EduMod-LLM: A Modular Approach for Designing Flexible and Transparent Educational Assistantscited by 1
  35. 2025A Comparative Study of Student Perspectives on Technical Writing Feedback Quality: Evaluating LLMs, SLMs, and Humans in Computer Science Topicscited by 1
  36. 2026Evaluating adaptive and generative AI-based feedback and recommendations in a knowledge-graph-integrated programming learning systemcited by 1
  37. 2026Generative Experiences for Digital Mental Health Interventions: Evidence from a Randomized Studycited by 1
  38. 2026A Scoping Review of Large Language Model-Based Pedagogical Agentscited by 1

50 most cited of 81 All 81 (JSON)