mlsanityData to 7 Oct 2026

A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education

arXiv 2312.03173 ↗First posted 5 Dec 2023cs.CY cs.AI cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

There is a constant need for educators to develop and maintain effective up-to-date assessments. While there is a growing body of research in computing education on utilizing large language models (LLMs) in generation and engagement with coding exercises, the use of LLMs for generating programming MCQs has not been extensively explored. We analyzed the capability of GPT-4 to produce multiple-choice questions (MCQs) aligned with specific learning objectives (LOs) from Python programming classes in higher education. Specifically, we developed an LLM-powered (GPT-4) system for generation of MCQs from high-level course context and module-level LOs. We evaluated 651 LLM-generated and 449 human-crafted MCQs aligned to 246 LOs from 6 Python courses. We found that GPT-4 was capable of producing MCQs with clear language, a single correct choice, and high-quality distractors. We also observed that the generated MCQs appeared to be well-aligned with the LOs. Our findings can be leveraged by educators wishing to take advantage of the state-of-the-art generative models to support MCQ authoring efforts.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 21 of its 61 references that are in the map. First posted on arXiv 5 Dec 2023.

Builds on

21 of 61 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Learning to Ask: Neural Question Generation for Reading Comprehensioncited by 186
  3. 2022Automatic Generation of Programming Exercises and Code Explanations using Large Language Modelscited by 124
  4. 2023Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programmingcited by 87
  5. 2023The Robots are Here: Navigating the Generative AI Revolution in Computing Educationcited by 85
  6. 2022Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Bookcited by 84
  7. 2022Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Languagecited by 81
  8. 2023Comparing Code Explanations Created by Students and Large Language Modelscited by 63
  9. 2023Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Coursescited by 38
  10. 2017Question Answering and Question Generation as Dual Taskscited by 37
  11. 2023Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Codecited by 31
  12. 2023Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?cited by 30
9 more
  1. 2018Generating Distractors for Reading Comprehension Questions from Real Examinationscited by 24
  2. 2020Knowledge-Driven Distractor Generation for Cloze-style Multiple Choice Questionscited by 24
  3. 2020Better Distractions: Transformer-based Distractor Generation and Multiple Choice Question Filteringcited by 17
  4. 2019Contrastive Multi-document Question Generationcited by 15
  5. 2019Co-Attention Hierarchical Network: Generating Coherent Long Distractors for Reading Comprehensioncited by 14
  6. 2023Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistantcited by 13
  7. 2021BERT-based distractor generation for Swedish reading comprehension questions using a small-scale datasetcited by 12
  8. 2023Prototyping the use of Large Language Models (LLMs) for adult learning content creation at scalecited by 12
  9. 2023From GPT-3 to GPT-4: On the Evolving Efficacy of LLMs to Answer Multiple-choice Questions for Programming Classes in Higher Educationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 11; 2026: 5.

34 papers.

Sort:

By area:

  1. 2024Large Language Models for Education: A Survey and Outlookcited by 185
  2. 2024Distractor Generation in Multiple-Choice Tasks: A Survey of Methods, Datasets, and Evaluationcited by 25
  3. 2024ClimaQA: An Automated Evaluation Framework for Climate Question Answering Modelscited by 21
  4. 2024Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Toolscited by 13
  5. 2024Large Language Models in Computer Science Education: A Systematic Literature Reviewcited by 12
  6. 2024Unsupervised Distractor Generation via Large Language Model Distilling and Counterfactual Contrastive Decodingcited by 11
  7. 2024An Automatic Question Usability Evaluation Toolkitcited by 9
  8. 2025Generating Plausible Distractors for Multiple-Choice Questions via Student Choice Predictioncited by 8
  9. 2025Assessing AI-Generated Questions' Alignment with Cognitive Frameworks in Educational Assessmentcited by 8
  10. 2024Synthetic Students: A Comparative Study of Bug Distribution Between Large Language Models and Computing Studentscited by 6
  11. 2024Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programmingcited by 6
  12. 2024Generating AI Literacy MCQs: A Multi-Agent LLM Approachcited by 6
22 more
  1. 2024Understanding the Role of Temperature in Diverse Question Generation by GPT-4cited by 5
  2. 2025SlideBot: A Multi-Agent Framework for Generating Informative, Reliable, Multi-Modal Presentationscited by 4
  3. 2024Breaking the Programming Language Barrier: Multilingual Prompting to Empower Non-Native English Learnerscited by 2
  4. 2025A Study on Educational Data Analysis and Personalized Feedback Report Generation Based on Tags and ChatGPTcited by 2
  5. 2025Enhancing Student Learning with LLM-Generated Retrieval Practice Questions: An Empirical Study in Data Science Coursescited by 2
  6. 2024QACP: An Annotated Question Answering Dataset for Assisting Chinese Python Programming Learnerscited by 1
  7. 2024Comparative Analysis Vision of Worldwide AI Coursescited by 1
  8. 2024Evaluating the capability of large language models to personalize science texts for diverse middle-school-age learnerscited by 1
  9. 2024The use of GPT-4o and Other Large Language Models for the Improvement and Design of Self-Assessment Scales for Measurement of Interpersonal Communication Skillscited by 1
  10. 2024Easy Come, Easy Go? Examining the Perceptions and Learning Effects of LLM-based Chatbot in the Context of Search-as-Learningcited by 1
  11. 2025Prompt Programming: A Platform for Dialogue-based Computational Problem Solving with Generative AI Modelscited by 1
  12. 2025Generate-Then-Validate: A Novel Question Generation Approach Using Small Language Modelscited by 1
  13. 2026Automatic Question Generation for Intuitive Learning Utilizing Causal Graph Guided Chain of Thought Reasoningcited by 1
  14. 2026Automatic Classification of Pedagogical Materials against CS Curriculum Guidelinescited by 1
  15. 2026CODE-GEN: A Human-in-the-Loop RAG-Based Agentic AI System for Multiple-Choice Question Generationcited by 1
  16. 2024Automatic Item Generation for Personality Situational Judgment Tests with Large Language Modelscited by 0
  17. 2025Slide2Text: Leveraging LLMs for Personalized Textbook Generation from PowerPoint Presentationscited by 0
  18. 2025Harnessing Structured Knowledge: A Concept Map-Based Approach for High-Quality Multiple Choice Question Generation with Effective Distractorscited by 0
  19. 2025LingoQ: Bridging the Gap between EFL Learning and Work through AI-Generated Work-Related Quizzescited by 0
  20. 2025From Pilots to Practices: A Scoping Review of GenAI-Enabled Personalization in Computer Science Educationcited by 0
  21. 2026Hallucination-Free Automatic Question & Answer Generation for Intuitive Learningcited by 0
  22. 2026The Architect, the Adversary, and the Judge: Closed-Loop Generation of Standards-Aligned Assessment Items at Scalecited by 0