mlsanityData to 14 Oct 2026

Automatic Generation of Programming Exercises and Code Explanations using Large Language Models

arXiv 2206.11861 ↗First posted 3 Jun 2022cs.SE cs.AI cs.CLcited by 126

Not on the map. Why?

Abstract (arXiv)

This article explores the natural language generation capabilities of large language models with application to the production of two types of learning resources common in programming courses. Using OpenAI Codex as the large language model, we create programming exercises (including sample solutions and test cases) and code explanations, assessing these qualitatively and quantitatively. Our results suggest that the majority of the automatically generated content is both novel and sensible, and in some cases ready to use as is. When creating exercises we find that it is remarkably easy to influence both the programming concepts and the contextual themes they contain, simply by supplying keywords as input to the model. Our analysis suggests that there is significant value in massive generative machine learning models as a tool for instructors, although there remains a need for some oversight to ensure the quality of the generated content before it is delivered to students. We further discuss the implications of OpenAI Codex and similar tools for introductory programming education and highlight future research streams that have the potential to improve the quality of the educational experience for both teachers and students alike.

Key facts. Cited by 126 papers in the map (data through 14 Oct 2026). Builds on 10 of its 103 references that are in the map. First posted on arXiv 3 Jun 2022.

Builds on

10 of 103 references.

References · 10

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  5. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  6. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  7. 2021A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Levelcited by 96
  8. 2022Pop Quiz! Can a Large Language Model Help With Reverse Engineering?cited by 14
  9. 2022To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set?cited by 11
  10. 2021Let's Ask Students About Their Programs, Automaticallycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 47; 2024: 45; 2025: 19; 2026: 10.

126 papers.

Sort:

By area:

  1. 2022CodeT: Code Generation with Generated Testscited by 384
  2. 2023Using an LLM to Help With Code Understandingcited by 228
  3. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  4. 2022Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generationcited by 99
  5. 2023The Robots are Here: Navigating the Generative AI Revolution in Computing Educationcited by 86
  6. 2024CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needscited by 84
  7. 2023Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Reviewcited by 83
  8. 2022Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Languagecited by 82
  9. 2022Language Models Can Teach Themselves to Program Bettercited by 78
  10. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 76
  11. 2024Building Machines that Learn and Think with Peoplecited by 73
  12. 2023CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classescited by 72
38 more
  1. 2022Using Large Language Models to Enhance Programming Error Messagescited by 67
  2. 2023"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmerscited by 64
  3. 2023Comparing Code Explanations Created by Students and Large Language Modelscited by 64
  4. 2023Computing Education in the Era of Generative AIcited by 52
  5. 2024The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmerscited by 48
  6. 2023Exploring the Responses of Large Language Models to Beginner Programmers' Help Requestscited by 46
  7. 2023Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Coursescited by 39
  8. 2023Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutorscited by 39
  9. 2023A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Educationcited by 35
  10. 2023Generating High-Precision Feedback for Programming Syntax Errors using Large Language Modelscited by 34
  11. 2023Detecting LLM-Generated Text in Computing Education: A Comparative Study for ChatGPT Casescited by 32
  12. 2023Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validationcited by 29
  13. 2025Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubricscited by 29
  14. 2023Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Modelscited by 28
  15. 2023Exploring the Potential of Large Language Models to Generate Formative Programming Feedbackcited by 28
  16. 2023Refactoring Programs Using Large Language Models with Few-Shot Examplescited by 28
  17. 2023Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generationcited by 22
  18. 2024The Landscape and Challenges of HPC Research and LLMscited by 22
  19. 2023Next-Step Hint Generation for Introductory Programming Using Large Language Modelscited by 20
  20. 2023AI-native Interconnect Framework for Integration of Large Language Model Technologies in 6G Systemscited by 19
  21. 2025LAMD: Context-driven Android Malware Detection and Classification with LLMscited by 18
  22. 2023Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generatorscited by 17
  23. 2023Exploring the Robustness of Large Language Models for Solving Programming Problemscited by 16
  24. 2023Harnessing LLMs in Curricular Design: Using GPT-4 to Support Authoring of Learning Objectivescited by 16
  25. 2024Improving Socratic Question Generation using Data Augmentation and Preference Optimizationcited by 16
  26. 2024Feedback-Generation for Programming Exercises With GPT-4cited by 14
  27. 2024Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Toolscited by 14
  28. 2023ChatGPT in the Classroom: An Analysis of Its Strengths and Weaknesses for Solving Undergraduate Computer Science Questionscited by 13
  29. 2023Neural Task Synthesis for Visual Programmingcited by 13
  30. 2023Conceptual Design Generation Using Large Language Modelscited by 13
  31. 2023Evaluating ChatGPT and GPT-4 for Visual Programmingcited by 13
  32. 2024Evaluating Contextually Personalized Programming Exercises Created with Generative AIcited by 13
  33. 2024Large Language Models in Computer Science Education: A Systematic Literature Reviewcited by 13
  34. 2023Can We Trust AI-Generated Educational Content? Comparative Analysis of Human and AI-Generated Learning Resourcescited by 12
  35. 2024Evaluating and Optimizing Educational Content with Large Language Model Judgmentscited by 12
  36. 2023Evoke: Evoking Critical Thinking Abilities in LLMs via Reviewer-Author Prompt Editingcited by 11
  37. 2023More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problemscited by 11
  38. 2024How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Coursecited by 11

50 most cited of 126 All 126 (JSON)