# A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education

arXiv: https://arxiv.org/abs/2312.03173 · HTML: https://mlsanity.com/paper/2312.03173/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 61 references that are in the map.
- First posted on arXiv 5 Dec 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning to Ask: Neural Question Generation for Reading Comprehension](https://mlsanity.com/paper/1705.00106/) (2017)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://mlsanity.com/paper/2302.07427/) (2023)
- [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023)
- [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022)
- [Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language](https://mlsanity.com/paper/2210.15157/) (2022)
- [Comparing Code Explanations Created by Students and Large Language Models](https://mlsanity.com/paper/2304.03938/) (2023)
- [Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses](https://mlsanity.com/paper/2306.10073/) (2023)
- [Question Answering and Question Generation as Dual Tasks](https://mlsanity.com/paper/1706.02027/) (2017)
- [Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code](https://mlsanity.com/paper/2303.08033/) (2023)
- [Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?](https://mlsanity.com/paper/2306.13906/) (2023)
- [Generating Distractors for Reading Comprehension Questions from Real Examinations](https://mlsanity.com/paper/1809.02768/) (2018)
- [Knowledge-Driven Distractor Generation for Cloze-style Multiple Choice Questions](https://mlsanity.com/paper/2004.09853/) (2020)
- [Better Distractions: Transformer-based Distractor Generation and Multiple Choice Question Filtering](https://mlsanity.com/paper/2010.09598/) (2020)
- [Contrastive Multi-document Question Generation](https://mlsanity.com/paper/1911.03047/) (2019)
- [Co-Attention Hierarchical Network: Generating Coherent Long Distractors for Reading Comprehension](https://mlsanity.com/paper/1911.08648/) (2019)
- [Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant](https://mlsanity.com/paper/2310.16984/) (2023)
- [BERT-based distractor generation for Swedish reading comprehension questions using a small-scale dataset](https://mlsanity.com/paper/2108.03973/) (2021)
- [Prototyping the use of Large Language Models (LLMs) for adult learning content creation at scale](https://mlsanity.com/paper/2306.01815/) (2023)
- [From GPT-3 to GPT-4: On the Evolving Efficacy of LLMs to Answer Multiple-choice Questions for Programming Classes in Higher Education](https://mlsanity.com/paper/2311.09518/) (2023)

## Built on by (most cited first, 30 of 35)

- [Large Language Models for Education: A Survey and Outlook](https://mlsanity.com/paper/2403.18105/) (2024)
- [Distractor Generation in Multiple-Choice Tasks: A Survey of Methods, Datasets, and Evaluation](https://mlsanity.com/paper/2402.01512/) (2024)
- [ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models](https://mlsanity.com/paper/2410.16701/) (2024)
- [Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools](https://mlsanity.com/paper/2412.14732/) (2024)
- [Large Language Models in Computer Science Education: A Systematic Literature Review](https://mlsanity.com/paper/2410.16349/) (2024)
- [Unsupervised Distractor Generation via Large Language Model Distilling and Counterfactual Contrastive Decoding](https://mlsanity.com/paper/2406.01306/) (2024)
- [An Automatic Question Usability Evaluation Toolkit](https://mlsanity.com/paper/2405.20529/) (2024)
- [Generating Plausible Distractors for Multiple-Choice Questions via Student Choice Prediction](https://mlsanity.com/paper/2501.13125/) (2025)
- [Assessing AI-Generated Questions' Alignment with Cognitive Frameworks in Educational Assessment](https://mlsanity.com/paper/2504.14232/) (2025)
- [Synthetic Students: A Comparative Study of Bug Distribution Between Large Language Models and Computing Students](https://mlsanity.com/paper/2410.09193/) (2024)
- [Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming](https://mlsanity.com/paper/2411.09261/) (2024)
- [Generating AI Literacy MCQs: A Multi-Agent LLM Approach](https://mlsanity.com/paper/2412.00970/) (2024)
- [Understanding the Role of Temperature in Diverse Question Generation by GPT-4](https://mlsanity.com/paper/2404.09366/) (2024)
- [SlideBot: A Multi-Agent Framework for Generating Informative, Reliable, Multi-Modal Presentations](https://mlsanity.com/paper/2511.09804/) (2025)
- [Breaking the Programming Language Barrier: Multilingual Prompting to Empower Non-Native English Learners](https://mlsanity.com/paper/2412.12800/) (2024)
- [A Study on Educational Data Analysis and Personalized Feedback Report Generation Based on Tags and ChatGPT](https://mlsanity.com/paper/2501.06819/) (2025)
- [Prompt Programming: A Platform for Dialogue-based Computational Problem Solving with Generative AI Models](https://mlsanity.com/paper/2503.04267/) (2025)
- [Enhancing Student Learning with LLM-Generated Retrieval Practice Questions: An Empirical Study in Data Science Courses](https://mlsanity.com/paper/2507.05629/) (2025)
- [QACP: An Annotated Question Answering Dataset for Assisting Chinese Python Programming Learners](https://mlsanity.com/paper/2402.07913/) (2024)
- [Comparative Analysis Vision of Worldwide AI Courses](https://mlsanity.com/paper/2407.16881/) (2024)
- [Evaluating the capability of large language models to personalize science texts for diverse middle-school-age learners](https://mlsanity.com/paper/2408.05204/) (2024)
- [The use of GPT-4o and Other Large Language Models for the Improvement and Design of Self-Assessment Scales for Measurement of Interpersonal Communication Skills](https://mlsanity.com/paper/2409.14050/) (2024)
- [Easy Come, Easy Go? Examining the Perceptions and Learning Effects of LLM-based Chatbot in the Context of Search-as-Learning](https://mlsanity.com/paper/2410.01396/) (2024)
- [Generate-Then-Validate: A Novel Question Generation Approach Using Small Language Models](https://mlsanity.com/paper/2512.10110/) (2025)
- [Automatic Question Generation for Intuitive Learning Utilizing Causal Graph Guided Chain of Thought Reasoning](https://mlsanity.com/paper/2601.06098/) (2026)
- [Automatic Classification of Pedagogical Materials against CS Curriculum Guidelines](https://mlsanity.com/paper/2602.03962/) (2026)
- [CODE-GEN: A Human-in-the-Loop RAG-Based Agentic AI System for Multiple-Choice Question Generation](https://mlsanity.com/paper/2604.03926/) (2026)
- [Automatic Item Generation for Personality Situational Judgment Tests with Large Language Models](https://mlsanity.com/paper/2412.12144/) (2024)
- [Slide2Text: Leveraging LLMs for Personalized Textbook Generation from PowerPoint Presentations](https://mlsanity.com/paper/2503.17710/) (2025)
- [Harnessing Structured Knowledge: A Concept Map-Based Approach for High-Quality Multiple Choice Question Generation with Effective Distractors](https://mlsanity.com/paper/2505.02850/) (2025)

## Abstract (arXiv)

There is a constant need for educators to develop and maintain effective up-to-date assessments. While there is a growing body of research in computing education on utilizing large language models (LLMs) in generation and engagement with coding exercises, the use of LLMs for generating programming MCQs has not been extensively explored. We analyzed the capability of GPT-4 to produce multiple-choice questions (MCQs) aligned with specific learning objectives (LOs) from Python programming classes in higher education. Specifically, we developed an LLM-powered (GPT-4) system for generation of MCQs from high-level course context and module-level LOs. We evaluated 651 LLM-generated and 449 human-crafted MCQs aligned to 246 LOs from 6 Python courses. We found that GPT-4 was capable of producing MCQs with clear language, a single correct choice, and high-quality distractors. We also observed that the generated MCQs appeared to be well-aligned with the LOs. Our findings can be leveraged by educators wishing to take advantage of the state-of-the-art generative models to support MCQ authoring efforts.