mlsanityData to 14 Oct 2026

The Robots are Here: Navigating the Generative AI Revolution in Computing Education

arXiv 2310.00658 ↗First posted 1 Oct 2023cs.CY cs.AI cs.HCcited by 86

Not on the map. Why?

Abstract (arXiv)

Recent advancements in artificial intelligence (AI) are fundamentally reshaping computing, with large language models (LLMs) now effectively being able to generate and interpret source code and natural language instructions. These emergent capabilities have sparked urgent questions in the computing education community around how educators should adapt their pedagogy to address the challenges and to leverage the opportunities presented by this new technology. In this working group report, we undertake a comprehensive exploration of LLMs in the context of computing education and make five significant contributions. First, we provide a detailed review of the literature on LLMs in computing education and synthesise findings from 71 primary articles. Second, we report the findings of a survey of computing students and instructors from across 20 countries, capturing prevailing attitudes towards LLMs and their use in computing education contexts. Third, to understand how pedagogy is already changing, we offer insights collected from in-depth interviews with 22 computing educators from five continents who have already adapted their curricula and assessments. Fourth, we use the ACM Code of Ethics to frame a discussion of ethical issues raised by the use of large language models in computing education, and we provide concrete advice for policy makers, educators, and students. Finally, we benchmark the performance of LLMs on various computing education datasets, and highlight the extent to which the capabilities of current models are rapidly improving. Our aim is that this report will serve as a focal point for both researchers and practitioners who are exploring, adapting, using, and evaluating LLMs and LLM-based tools in computing classrooms.

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). Builds on 64 of its 178 references that are in the map. First posted on arXiv 1 Oct 2023.

Builds on

64 of 178 references.

References · 64

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  6. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  7. 2021Measuring Coding Challenge Competence With APPScited by 902
  8. 2023Using an LLM to Help With Code Understandingcited by 228
  9. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 218
  10. 2019SPoC: Search-based Pseudocode to Codecited by 176
  11. 2021The Values Encoded in Machine Learning Researchcited by 172
  12. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 152
38 more
  1. 2023Co-Writing with Opinionated Language Models Affects Users' Viewscited by 148
  2. 2022Large Language Models Meet NL2Code: A Surveycited by 138
  3. 2022Automatic Generation of Programming Exercises and Code Explanations using Large Language Modelscited by 126
  4. 2022Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generationcited by 99
  5. 2022Lost at C: A User Study on the Security Implications of Large Language Model Code Assistantscited by 91
  6. 2023Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programmingcited by 89
  7. 2022Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Bookcited by 84
  8. 2022Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Languagecited by 82
  9. 2023CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classescited by 72
  10. 2022Using Large Language Models to Enhance Programming Error Messagescited by 67
  11. 2023"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmerscited by 64
  12. 2023Comparing Code Explanations Created by Students and Large Language Modelscited by 64
  13. 2023Computing Education in the Era of Generative AIcited by 52
  14. 2023Exploring the Responses of Large Language Models to Beginner Programmers' Help Requestscited by 46
  15. 2023AI-assisted coding: Experiments with GPT-4cited by 43
  16. 2023ChatGPT and Software Testing Education: Promises & Perilscited by 42
  17. 2023Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Coursescited by 39
  18. 2023Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutorscited by 39
  19. 2023Generating High-Precision Feedback for Programming Syntax Errors using Large Language Modelscited by 34
  20. 2023A Comprehensive AI Policy Education Framework for University Teaching and Learningcited by 34
  21. 2023Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Codecited by 32
  22. 2023StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Codecited by 32
  23. 2023Large Language Models (GPT) for automating feedback on programming assignmentscited by 32
  24. 2023Detecting LLM-Generated Text in Computing Education: A Comparative Study for ChatGPT Casescited by 32
  25. 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 31
  26. 2023Exploring the Potential of Large Language Models to Generate Formative Programming Feedbackcited by 28
  27. 2023A large language model-assisted education tool to provide feedback on open-ended responsescited by 22
  28. 2023Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programmingcited by 20
  29. 2023Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generatorscited by 17
  30. 2023Generative AI in Computing Education: Perspectives of Students and Instructorscited by 17
  31. 2023Harnessing LLMs in Curricular Design: Using GPT-4 to Support Authoring of Learning Objectivescited by 16
  32. 2023Successful Combination of Database Search and Snowballing for Identification of Primary Studies in Systematic Literature Studiescited by 16
  33. 2022AI-driven Development Is Here: Should You Worry?cited by 15
  34. 2023Neural Task Synthesis for Visual Programmingcited by 13
  35. 2023Evaluating ChatGPT and GPT-4 for Visual Programmingcited by 13
  36. 2022Automatically Generating CS Learning Materials with Large Language Modelscited by 12
  37. 2023Many bioinformatics programming tasks can be automated with ChatGPTcited by 12
  38. 2023The AI generation gap: Are Gen Z students more interested in adopting generative AI such as ChatGPT in teaching and learning than their Gen X and Millennial Generation teachers?cited by 12

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 38; 2025: 25; 2026: 17.

86 papers.

Sort:

By area:

  1. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  2. 2024The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmerscited by 48
  3. 2023A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Educationcited by 35
  4. 2023Next-Step Hint Generation for Introductory Programming Using Large Language Modelscited by 20
  5. 2024Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judgecited by 20
  6. 2024Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learningcited by 16
  7. 2024Feedback-Generation for Programming Exercises With GPT-4cited by 14
  8. 2024Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Toolscited by 14
  9. 2024Generative AI for Education (GAIED): Advances, Opportunities, and Challengescited by 13
  10. 2024Evaluating Contextually Personalized Programming Exercises Created with Generative AIcited by 13
  11. 2024Large Language Models in Computer Science Education: A Systematic Literature Reviewcited by 13
  12. 2024Exploring the Capabilities of Prompted Large Language Models in Educational and Assessment Applicationscited by 12
38 more
  1. 2023More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problemscited by 11
  2. 2023Generative AI in Higher Education: Seeing ChatGPT Through Universities' Policies, Resources, and Guidelinescited by 11
  3. 2024How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Coursecited by 11
  4. 2024Evaluating the Application of Large Language Models to Generate Feedback in Programming Educationcited by 10
  5. 2024Generative AI in Education: A Study of Educators' Awareness, Sentiments, and Influencing Factorscited by 10
  6. 2025How Do Programming Students Use Generative AI?cited by 10
  7. 2024Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Coursecited by 9
  8. 2024Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPTcited by 8
  9. 2024Evaluating Language Models for Generating and Judging Programming Feedbackcited by 8
  10. 2024Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generationcited by 7
  11. 2023Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Modelscited by 6
  12. 2024MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answeringcited by 6
  13. 2024"ChatGPT Is Here to Help, Not to Replace Anybody" -- An Evaluation of Students' Opinions On Integrating ChatGPT In CS Coursescited by 6
  14. 2024Leveraging Lecture Content for Improved Feedback: Explorations with GPT-4 and Retrieval Augmented Generationcited by 6
  15. 2024Students' Perceptions and Use of Generative AI Tools for Programming Across Different Computing Coursescited by 6
  16. 2024Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programmingcited by 6
  17. 2024"Like a Nesting Doll": Analyzing Recursion Analogies Generated by CS Students using Large Language Modelscited by 5
  18. 2026Evaluating Theory of Mind and Internal Beliefs in LLM-Based Multi-Agent Systemscited by 5
  19. 2023Efficient Classification of Student Help Requests in Programming Courses Using Large Language Modelscited by 4
  20. 2024Education in the Era of Neurosymbolic AIcited by 4
  21. 2024The Evolving Usage of GenAI by Computing Studentscited by 4
  22. 2025AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Reviewcited by 4
  23. 2024Enhancing Programming Error Messages in Real Time with Generative AIcited by 3
  24. 2024One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Taskscited by 3
  25. 2025Novice Developers' Perspectives on Adopting LLMs for Software Development: A Systematic Literature Reviewcited by 3
  26. 2025Synthesizing High-Quality Programming Tasks with LLM-based Expert and Student Agentscited by 3
  27. 2025Exploring Student Behaviors and Motivations using AI TAs with Optional Guardrailscited by 3
  28. 2024Understanding Help-Seeking Behavior of Students Using LLMs vs. Web Search for Writing SQL Queriescited by 2
  29. 2024Seeing the Forest and the Trees: Solving Visual Graph and Tree Based Data Structure Problems using Large Multimodal Modelscited by 2
  30. 2024Breaking the Programming Language Barrier: Multilingual Prompting to Empower Non-Native English Learnerscited by 2
  31. 2025The Role of Generative AI in Software Student CollaborAItioncited by 2
  32. 2025Prompt Programming: A Platform for Dialogue-based Computational Problem Solving with Generative AI Modelscited by 2
  33. 2025Toward Personalizing Quantum Computing Education: An Evolutionary LLM-Powered Approachcited by 2
  34. 2025Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Educationcited by 2
  35. 2025Scaffolding Metacognition in Programming Education: Understanding Student-AI Interactions and Design Implicationscited by 2
  36. 2024A Survey Study on the State of the Art of Programming Exercise Generation using Large Language Modelscited by 1
  37. 2024Integrating Natural Language Prompting Tasks in Introductory Programming Coursescited by 1
  38. 2024On the Opportunities of Large Language Models for Programming Process Datacited by 1

50 most cited of 86 All 86 (JSON)