The Robots are Here: Navigating the Generative AI Revolution in Computing Education
Not on the map. Why?
Abstract (arXiv)
Recent advancements in artificial intelligence (AI) are fundamentally reshaping computing, with large language models (LLMs) now effectively being able to generate and interpret source code and natural language instructions. These emergent capabilities have sparked urgent questions in the computing education community around how educators should adapt their pedagogy to address the challenges and to leverage the opportunities presented by this new technology. In this working group report, we undertake a comprehensive exploration of LLMs in the context of computing education and make five significant contributions. First, we provide a detailed review of the literature on LLMs in computing education and synthesise findings from 71 primary articles. Second, we report the findings of a survey of computing students and instructors from across 20 countries, capturing prevailing attitudes towards LLMs and their use in computing education contexts. Third, to understand how pedagogy is already changing, we offer insights collected from in-depth interviews with 22 computing educators from five continents who have already adapted their curricula and assessments. Fourth, we use the ACM Code of Ethics to frame a discussion of ethical issues raised by the use of large language models in computing education, and we provide concrete advice for policy makers, educators, and students. Finally, we benchmark the performance of LLMs on various computing education datasets, and highlight the extent to which the capabilities of current models are rapidly improving. Our aim is that this report will serve as a focal point for both researchers and practitioners who are exploring, adapting, using, and evaluating LLMs and LLM-based tools in computing classrooms.
Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). Builds on 64 of its 178 references that are in the map. First posted on arXiv 1 Oct 2023.
Builds on
64 of 178 references.
References · 64
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2022Survey of Hallucination in Natural Language Generationcited by 2,075
- 2022Competition-Level Code Generation with AlphaCodecited by 1,462
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
- 2021Measuring Coding Challenge Competence With APPScited by 902
- 2023Using an LLM to Help With Code Understandingcited by 228
- 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 218
- 2019SPoC: Search-based Pseudocode to Codecited by 176
- 2021The Values Encoded in Machine Learning Researchcited by 172
- 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 152
38 more
- 2023Co-Writing with Opinionated Language Models Affects Users' Viewscited by 148
- 2022Large Language Models Meet NL2Code: A Surveycited by 138
- 2022Automatic Generation of Programming Exercises and Code Explanations using Large Language Modelscited by 126
- 2022Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generationcited by 99
- 2022Lost at C: A User Study on the Security Implications of Large Language Model Code Assistantscited by 91
- 2023Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programmingcited by 89
- 2022Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Bookcited by 84
- 2022Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Languagecited by 82
- 2023CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classescited by 72
- 2022Using Large Language Models to Enhance Programming Error Messagescited by 67
- 2023"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmerscited by 64
- 2023Comparing Code Explanations Created by Students and Large Language Modelscited by 64
- 2023Computing Education in the Era of Generative AIcited by 52
- 2023Exploring the Responses of Large Language Models to Beginner Programmers' Help Requestscited by 46
- 2023AI-assisted coding: Experiments with GPT-4cited by 43
- 2023ChatGPT and Software Testing Education: Promises & Perilscited by 42
- 2023Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Coursescited by 39
- 2023Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutorscited by 39
- 2023Generating High-Precision Feedback for Programming Syntax Errors using Large Language Modelscited by 34
- 2023A Comprehensive AI Policy Education Framework for University Teaching and Learningcited by 34
- 2023Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Codecited by 32
- 2023StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Codecited by 32
- 2023Large Language Models (GPT) for automating feedback on programming assignmentscited by 32
- 2023Detecting LLM-Generated Text in Computing Education: A Comparative Study for ChatGPT Casescited by 32
- 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 31
- 2023Exploring the Potential of Large Language Models to Generate Formative Programming Feedbackcited by 28
- 2023A large language model-assisted education tool to provide feedback on open-ended responsescited by 22
- 2023Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programmingcited by 20
- 2023Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generatorscited by 17
- 2023Generative AI in Computing Education: Perspectives of Students and Instructorscited by 17
- 2023Harnessing LLMs in Curricular Design: Using GPT-4 to Support Authoring of Learning Objectivescited by 16
- 2023Successful Combination of Database Search and Snowballing for Identification of Primary Studies in Systematic Literature Studiescited by 16
- 2022AI-driven Development Is Here: Should You Worry?cited by 15
- 2023Neural Task Synthesis for Visual Programmingcited by 13
- 2023Evaluating ChatGPT and GPT-4 for Visual Programmingcited by 13
- 2022Automatically Generating CS Learning Materials with Large Language Modelscited by 12
- 2023Many bioinformatics programming tasks can be automated with ChatGPTcited by 12
- 2023The AI generation gap: Are Gen Z students more interested in adopting generative AI such as ChatGPT in teaching and learning than their Gen X and Millennial Generation teachers?cited by 12
50 most cited of 64 All 64 (JSON)
Built on by
Citing papers per year: 2023: 6; 2024: 38; 2025: 25; 2026: 17.
86 papers.
Sort:
By area:
- 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
- 2024The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmerscited by 48
- 2023A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Educationcited by 35
- 2023Next-Step Hint Generation for Introductory Programming Using Large Language Modelscited by 20
- 2024Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judgecited by 20
- 2024Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learningcited by 16
- 2024Feedback-Generation for Programming Exercises With GPT-4cited by 14
- 2024Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Toolscited by 14
- 2024Generative AI for Education (GAIED): Advances, Opportunities, and Challengescited by 13
- 2024Evaluating Contextually Personalized Programming Exercises Created with Generative AIcited by 13
- 2024Large Language Models in Computer Science Education: A Systematic Literature Reviewcited by 13
- 2024Exploring the Capabilities of Prompted Large Language Models in Educational and Assessment Applicationscited by 12
38 more
- 2023More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problemscited by 11
- 2023Generative AI in Higher Education: Seeing ChatGPT Through Universities' Policies, Resources, and Guidelinescited by 11
- 2024How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Coursecited by 11
- 2024Evaluating the Application of Large Language Models to Generate Feedback in Programming Educationcited by 10
- 2024Generative AI in Education: A Study of Educators' Awareness, Sentiments, and Influencing Factorscited by 10
- 2025How Do Programming Students Use Generative AI?cited by 10
- 2024Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Coursecited by 9
- 2024Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPTcited by 8
- 2024Evaluating Language Models for Generating and Judging Programming Feedbackcited by 8
- 2024Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generationcited by 7
- 2023Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Modelscited by 6
- 2024MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answeringcited by 6
- 2024"ChatGPT Is Here to Help, Not to Replace Anybody" -- An Evaluation of Students' Opinions On Integrating ChatGPT In CS Coursescited by 6
- 2024Leveraging Lecture Content for Improved Feedback: Explorations with GPT-4 and Retrieval Augmented Generationcited by 6
- 2024Students' Perceptions and Use of Generative AI Tools for Programming Across Different Computing Coursescited by 6
- 2024Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programmingcited by 6
- 2024"Like a Nesting Doll": Analyzing Recursion Analogies Generated by CS Students using Large Language Modelscited by 5
- 2026Evaluating Theory of Mind and Internal Beliefs in LLM-Based Multi-Agent Systemscited by 5
- 2023Efficient Classification of Student Help Requests in Programming Courses Using Large Language Modelscited by 4
- 2024Education in the Era of Neurosymbolic AIcited by 4
- 2024The Evolving Usage of GenAI by Computing Studentscited by 4
- 2025AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Reviewcited by 4
- 2024Enhancing Programming Error Messages in Real Time with Generative AIcited by 3
- 2024One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Taskscited by 3
- 2025Novice Developers' Perspectives on Adopting LLMs for Software Development: A Systematic Literature Reviewcited by 3
- 2025Synthesizing High-Quality Programming Tasks with LLM-based Expert and Student Agentscited by 3
- 2025Exploring Student Behaviors and Motivations using AI TAs with Optional Guardrailscited by 3
- 2024Understanding Help-Seeking Behavior of Students Using LLMs vs. Web Search for Writing SQL Queriescited by 2
- 2024Seeing the Forest and the Trees: Solving Visual Graph and Tree Based Data Structure Problems using Large Multimodal Modelscited by 2
- 2024Breaking the Programming Language Barrier: Multilingual Prompting to Empower Non-Native English Learnerscited by 2
- 2025The Role of Generative AI in Software Student CollaborAItioncited by 2
- 2025Prompt Programming: A Platform for Dialogue-based Computational Problem Solving with Generative AI Modelscited by 2
- 2025Toward Personalizing Quantum Computing Education: An Evolutionary LLM-Powered Approachcited by 2
- 2025Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Educationcited by 2
- 2025Scaffolding Metacognition in Programming Education: Understanding Student-AI Interactions and Design Implicationscited by 2
- 2024A Survey Study on the State of the Art of Programming Exercise Generation using Large Language Modelscited by 1
- 2024Integrating Natural Language Prompting Tasks in Introductory Programming Coursescited by 1
- 2024On the Opportunities of Large Language Models for Programming Process Datacited by 1
50 most cited of 86 All 86 (JSON)
- 2026Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams24 Sep 2026
- 2026AgentForge: An Immersive Role-Playing Platform for Learning Agentic Software Engineering4 Aug 2026
- 2026Educating the Agentic Engineer: Curricula, Collaboration, and Continuous Learning in the AI Era31 Jul 2026
- 2026Is Solving Better Than Evaluating GenAI Solutions?30 Jul 2026
- 2026Did Alice Do Wrong? Cross-Cultural Differences in Student Perceptions of Generative AI Use in University Computing Education22 Jul 2026
- 2026A Comparative Analysis of Institutional and Course Generative AI Policies within Higher Education: Implications for Instruction in Computing Education14 Jul 2026
- 2026To Tab or Not to Tab: Measuring Critical Engagement in AI Code Completion Tools Using Behavioral Signals and Attention Checks29 Jun 2026
- 2026Exploring the Value of Diverse LLM Explanations in Introductory Programming27 Jun 2026
- 2026AI-Assisted Help-Seeking Trajectories in Programming Education from an SRL-Informed Perspective22 Jun 2026
- 2026Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent14 May 2026
- 2026AI-Generated Slides: Are They Good? Can Students Tell?13 May 2026
- 2026Benchmarked Yet Not Measured -- Generative AI Should be Evaluated Against Real-World Utility7 May 2026
38 more
- 2026The Pedagogy of AI Mistakes: Fostering Higher-Order Thinking6 May 2026
- 2026Evaluating LLMs for Answering Student Questions in Introductory Programming Courses30 Mar 2026
- 2026Evaluating Theory of Mind and Internal Beliefs in LLM-Based Multi-Agent Systems24 Feb 2026
- 2026From Code-Centric to Concept-Centric: Teaching NLP with LLM-Assisted "Vibe Coding"2 Feb 2026
- 2026From Toil to Thought: Designing for Strategic Exploration and Responsible AI in Systematic Literature Reviews21 Jan 2026
- 2025Assessing the Software Security Comprehension of Large Language Models24 Dec 2025
- 2025A Comparative Study of Student Perspectives on Technical Writing Feedback Quality: Evaluating LLMs, SLMs, and Humans in Computer Science Topics1 Dec 2025
- 2025Scaffolding Metacognition in Programming Education: Understanding Student-AI Interactions and Design Implications6 Nov 2025
- 2025Evaluating LLMs for Career Guidance: Comparative Analysis of Computing Competency Recommendations Across Ten African Countries20 Oct 2025
- 2025GenAI Voice Mode in Programming Education12 Sep 2025
- 2025Detecting Struggling Student Programmers using Proficiency Taxonomies24 Aug 2025
- 2025Ambiguity Resolution with Human Feedback for Code Writing Tasks18 Aug 2025
- 2025Teaching Introduction to Programming in the times of AI: A case study of a course re-design7 Aug 2025
- 2025What Makes Code Generation Ethically Sourced?26 Jul 2025
- 2025Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?23 Jul 2025
- 2025Can AI support student engagement in classroom activities in higher education?22 Jun 2025
- 2025A Review of Generative AI in Computer Science Education: Challenges and Opportunities in Accuracy, Authenticity, and Assessment17 Jun 2025
- 2025Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education23 May 2025
- 2025Toward Personalizing Quantum Computing Education: An Evolutionary LLM-Powered Approach24 Apr 2025
- 2025Exploring Student Behaviors and Motivations using AI TAs with Optional Guardrails15 Apr 2025
- 2025"All Roads Lead to ChatGPT": How Generative AI is Eroding Social Interactions and Student Learning Communities14 Apr 2025
- 2025Synthesizing High-Quality Programming Tasks with LLM-based Expert and Student Agents10 Apr 2025
- 2025Novice Developers' Perspectives on Adopting LLMs for Software Development: A Systematic Literature Review10 Mar 2025
- 2025Prompt Programming: A Platform for Dialogue-based Computational Problem Solving with Generative AI Models6 Mar 2025
- 2025WIP: Assessing the Effectiveness of ChatGPT in Preparatory Testing Activities5 Mar 2025
- 2025AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review27 Feb 2025
- 2025The Role of Generative AI in Software Student CollaborAItion23 Jan 2025
- 2025To Google or To ChatGPT? A Comparison of CS2 Students' Information Gathering Approaches and Outcomes21 Jan 2025
- 2025How Do Programming Students Use Generative AI?17 Jan 2025
- 2025Lifelong Learning of Large Language Model based Agents: A Roadmap13 Jan 2025
- 2024The Evolving Usage of GenAI by Computing Students21 Dec 2024
- 2024Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools19 Dec 2024
- 2024Breaking the Programming Language Barrier: Multilingual Prompting to Empower Non-Native English Learners17 Dec 2024
- 2024Seeing the Forest and the Trees: Solving Visual Graph and Tree Based Data Structure Problems using Large Multimodal Models15 Dec 2024
- 2024Leveraging Large Language Models to Generate Course-specific Semantically Annotated Learning Objects5 Dec 2024
- 2024Education in the Era of Neurosymbolic AI16 Nov 2024
- 2024Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming14 Nov 2024
- 2024Towards Pedagogical LLMs with Supervised Fine Tuning for Computing Education4 Nov 2024
50 newest of 86 All 86 (JSON)
- 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
- 2026Evaluating Theory of Mind and Internal Beliefs in LLM-Based Multi-Agent Systemscited by 5
- 2026Educating the Agentic Engineer: Curricula, Collaboration, and Continuous Learning in the AI Eracited by 0
- 2026AgentForge: An Immersive Role-Playing Platform for Learning Agentic Software Engineeringcited by 0
- 2026AgentForge: An Immersive Role-Playing Platform for Learning Agentic Software Engineering4 Aug 2026
- 2026Educating the Agentic Engineer: Curricula, Collaboration, and Continuous Learning in the AI Era31 Jul 2026
- 2026Evaluating Theory of Mind and Internal Beliefs in LLM-Based Multi-Agent Systems24 Feb 2026
- 2025Lifelong Learning of Large Language Model based Agents: A Roadmap13 Jan 2025
- 2024Leveraging Lecture Content for Improved Feedback: Explorations with GPT-4 and Retrieval Augmented Generationcited by 6
- 2024Comparing the Utility, Preference, and Performance of Course Material Search Functionality and Retrieval-Augmented Generation Large Language Model (RAG-LLM) AI Chatbots in Information-Seeking Taskscited by 0
- 2024Comparing the Utility, Preference, and Performance of Course Material Search Functionality and Retrieval-Augmented Generation Large Language Model (RAG-LLM) AI Chatbots in Information-Seeking Tasks17 Oct 2024
- 2024Leveraging Lecture Content for Improved Feedback: Explorations with GPT-4 and Retrieval Augmented Generation5 May 2024
- 2024The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmerscited by 48
- 2023A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Educationcited by 35
- 2023Next-Step Hint Generation for Introductory Programming Using Large Language Modelscited by 20
- 2024Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judgecited by 20
- 2024Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learningcited by 16
- 2024Feedback-Generation for Programming Exercises With GPT-4cited by 14
- 2024Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Toolscited by 14
- 2024Generative AI for Education (GAIED): Advances, Opportunities, and Challengescited by 13
- 2024Evaluating Contextually Personalized Programming Exercises Created with Generative AIcited by 13
- 2024Large Language Models in Computer Science Education: A Systematic Literature Reviewcited by 13
- 2024Exploring the Capabilities of Prompted Large Language Models in Educational and Assessment Applicationscited by 12
- 2023Generative AI in Higher Education: Seeing ChatGPT Through Universities' Policies, Resources, and Guidelinescited by 11
12 most cited of 75 All 75 (JSON)
- 2026Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams24 Sep 2026
- 2026Is Solving Better Than Evaluating GenAI Solutions?30 Jul 2026
- 2026Did Alice Do Wrong? Cross-Cultural Differences in Student Perceptions of Generative AI Use in University Computing Education22 Jul 2026
- 2026A Comparative Analysis of Institutional and Course Generative AI Policies within Higher Education: Implications for Instruction in Computing Education14 Jul 2026
- 2026To Tab or Not to Tab: Measuring Critical Engagement in AI Code Completion Tools Using Behavioral Signals and Attention Checks29 Jun 2026
- 2026Exploring the Value of Diverse LLM Explanations in Introductory Programming27 Jun 2026
- 2026AI-Assisted Help-Seeking Trajectories in Programming Education from an SRL-Informed Perspective22 Jun 2026
- 2026Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent14 May 2026
- 2026AI-Generated Slides: Are They Good? Can Students Tell?13 May 2026
- 2026Benchmarked Yet Not Measured -- Generative AI Should be Evaluated Against Real-World Utility7 May 2026
- 2026The Pedagogy of AI Mistakes: Fostering Higher-Order Thinking6 May 2026
- 2026Evaluating LLMs for Answering Student Questions in Introductory Programming Courses30 Mar 2026
12 newest of 75 All 75 (JSON)