# The Robots are Here: Navigating the Generative AI Revolution in Computing Education

arXiv: https://arxiv.org/abs/2310.00658 · HTML: https://mlsanity.com/paper/2310.00658/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 178 references that are in the map.
- First posted on arXiv 1 Oct 2023.

## Builds on (most cited first, 30 of 64)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [SPoC: Search-based Pseudocode to Code](https://mlsanity.com/paper/1906.04908/) (2019)
- [The Values Encoded in Machine Learning Research](https://mlsanity.com/paper/2106.15590/) (2021)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [Co-Writing with Opinionated Language Models Affects Users' Views](https://mlsanity.com/paper/2302.00560/) (2023)
- [Large Language Models Meet NL2Code: A Survey](https://mlsanity.com/paper/2212.09420/) (2022)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation](https://mlsanity.com/paper/2212.01020/) (2022)
- [Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants](https://mlsanity.com/paper/2208.09727/) (2022)
- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://mlsanity.com/paper/2302.07427/) (2023)
- [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022)
- [Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language](https://mlsanity.com/paper/2210.15157/) (2022)
- [CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes](https://mlsanity.com/paper/2308.06921/) (2023)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- ["It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers](https://mlsanity.com/paper/2304.02491/) (2023)
- [Comparing Code Explanations Created by Students and Large Language Models](https://mlsanity.com/paper/2304.03938/) (2023)
- [Computing Education in the Era of Generative AI](https://mlsanity.com/paper/2306.02608/) (2023)
- [Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests](https://mlsanity.com/paper/2306.05715/) (2023)
- [AI-assisted coding: Experiments with GPT-4](https://mlsanity.com/paper/2304.13187/) (2023)
- [ChatGPT and Software Testing Education: Promises & Perils](https://mlsanity.com/paper/2302.03287/) (2023)
- [Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses](https://mlsanity.com/paper/2306.10073/) (2023)
- [Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors](https://mlsanity.com/paper/2306.17156/) (2023)

## Built on by (most cited first, 30 of 86)

- [Lifelong Learning of Large Language Model based Agents: A Roadmap](https://mlsanity.com/paper/2501.07278/) (2025)
- [The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers](https://mlsanity.com/paper/2405.17739/) (2024)
- [A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education](https://mlsanity.com/paper/2312.03173/) (2023)
- [Next-Step Hint Generation for Introductory Programming Using Large Language Models](https://mlsanity.com/paper/2312.10055/) (2023)
- [Open Source Language Models Can Provide Feedback: Evaluating LLMs' Ability to Help Students Using GPT-4-As-A-Judge](https://mlsanity.com/paper/2405.05253/) (2024)
- [Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learning](https://mlsanity.com/paper/2410.08922/) (2024)
- [Feedback-Generation for Programming Exercises With GPT-4](https://mlsanity.com/paper/2403.04449/) (2024)
- [Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools](https://mlsanity.com/paper/2412.14732/) (2024)
- [Generative AI for Education (GAIED): Advances, Opportunities, and Challenges](https://mlsanity.com/paper/2402.01580/) (2024)
- [Evaluating Contextually Personalized Programming Exercises Created with Generative AI](https://mlsanity.com/paper/2407.11994/) (2024)
- [Large Language Models in Computer Science Education: A Systematic Literature Review](https://mlsanity.com/paper/2410.16349/) (2024)
- [Exploring the Capabilities of Prompted Large Language Models in Educational and Assessment Applications](https://mlsanity.com/paper/2405.11579/) (2024)
- [More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problems](https://mlsanity.com/paper/2311.04926/) (2023)
- [Generative AI in Higher Education: Seeing ChatGPT Through Universities' Policies, Resources, and Guidelines](https://mlsanity.com/paper/2312.05235/) (2023)
- [How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Course](https://mlsanity.com/paper/2407.20792/) (2024)
- [Evaluating the Application of Large Language Models to Generate Feedback in Programming Education](https://mlsanity.com/paper/2403.09744/) (2024)
- [Generative AI in Education: A Study of Educators' Awareness, Sentiments, and Influencing Factors](https://mlsanity.com/paper/2403.15586/) (2024)
- [How Do Programming Students Use Generative AI?](https://mlsanity.com/paper/2501.10091/) (2025)
- [Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Course](https://mlsanity.com/paper/2406.06451/) (2024)
- [Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT](https://mlsanity.com/paper/2405.19132/) (2024)
- [Evaluating Language Models for Generating and Judging Programming Feedback](https://mlsanity.com/paper/2407.04873/) (2024)
- [Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation](https://mlsanity.com/paper/2406.05053/) (2024)
- [Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models](https://mlsanity.com/paper/2311.16017/) (2023)
- [MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answering](https://mlsanity.com/paper/2404.12926/) (2024)
- ["ChatGPT Is Here to Help, Not to Replace Anybody" -- An Evaluation of Students' Opinions On Integrating ChatGPT In CS Courses](https://mlsanity.com/paper/2404.17443/) (2024)
- [Leveraging Lecture Content for Improved Feedback: Explorations with GPT-4 and Retrieval Augmented Generation](https://mlsanity.com/paper/2405.06681/) (2024)
- [Students' Perceptions and Use of Generative AI Tools for Programming Across Different Computing Courses](https://mlsanity.com/paper/2410.06865/) (2024)
- [Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming](https://mlsanity.com/paper/2411.09261/) (2024)
- ["Like a Nesting Doll": Analyzing Recursion Analogies Generated by CS Students using Large Language Models](https://mlsanity.com/paper/2403.09409/) (2024)
- [Evaluating Theory of Mind and Internal Beliefs in LLM-Based Multi-Agent Systems](https://mlsanity.com/paper/2603.00142/) (2026)

## Abstract (arXiv)

Recent advancements in artificial intelligence (AI) are fundamentally reshaping computing, with large language models (LLMs) now effectively being able to generate and interpret source code and natural language instructions. These emergent capabilities have sparked urgent questions in the computing education community around how educators should adapt their pedagogy to address the challenges and to leverage the opportunities presented by this new technology. In this working group report, we undertake a comprehensive exploration of LLMs in the context of computing education and make five significant contributions. First, we provide a detailed review of the literature on LLMs in computing education and synthesise findings from 71 primary articles. Second, we report the findings of a survey of computing students and instructors from across 20 countries, capturing prevailing attitudes towards LLMs and their use in computing education contexts. Third, to understand how pedagogy is already changing, we offer insights collected from in-depth interviews with 22 computing educators from five continents who have already adapted their curricula and assessments. Fourth, we use the ACM Code of Ethics to frame a discussion of ethical issues raised by the use of large language models in computing education, and we provide concrete advice for policy makers, educators, and students. Finally, we benchmark the performance of LLMs on various computing education datasets, and highlight the extent to which the capabilities of current models are rapidly improving. Our aim is that this report will serve as a focal point for both researchers and practitioners who are exploring, adapting, using, and evaluating LLMs and LLM-based tools in computing classrooms.