mlsanityData to 14 Oct 2026

Exploring the Potential of Large Language Models in Generating Code-Tracing Questions for Introductory Programming Courses

arXiv 2310.15317 ↗First posted 23 Oct 2023cs.CL cs.CYcited by 3

Not on the map. Why?

Abstract (arXiv)

In this paper, we explore the application of large language models (LLMs) for generating code-tracing questions in introductory programming courses. We designed targeted prompts for GPT4, guiding it to generate code-tracing questions based on code snippets and descriptions. We established a set of human evaluation metrics to assess the quality of questions produced by the model compared to those created by human experts. Our analysis provides insights into the capabilities and potential of LLMs in generating diverse code-tracing questions. Additionally, we present a unique dataset of human and LLM-generated tracing questions, serving as a valuable resource for both the education and NLP research communities. This work contributes to the ongoing dialogue on the potential uses of LLMs in educational settings.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 22 of its 46 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

22 of 46 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  7. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  8. 2018CoQA: A Conversational Question Answering Challengecited by 683
  9. 2018QuAC : Question Answering in Contextcited by 381
  10. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  11. 2017Learning to Ask: Neural Question Generation for Reading Comprehensioncited by 186
  12. 2020Training Question Answering Models From Synthetic Datacited by 95
10 more
  1. 2023Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programmingcited by 89
  2. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 76
  3. 2023Using Large Language Models to Generate JUnit Tests: An Empirical Studycited by 53
  4. 2022Automatic Generation of Socratic Subquestions for Teaching Math Word Problemscited by 43
  5. 2020Semantic Graphs for Generating Deep Questionscited by 38
  6. 2020Unsupervised Multi-hop Question Answering by Question Generationcited by 38
  7. 2017Machine Comprehension by Text-to-Text Neural Question Generationcited by 32
  8. 2023Many bioinformatics programming tasks can be automated with ChatGPTcited by 12
  9. 2021Let's Ask Students About Their Programs, Automaticallycited by 3
  10. 2023Enhancing Programming eTextbooks with ChatGPT Generated Counterfactual-Thinking-Inspired Questionscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 1.

3 papers.

  1. 2024Large Language Models in Computer Science Education: A Systematic Literature Reviewcited by 13
  2. 2026What Drives Students' Use of AI Chatbots? Technology Acceptance in Conversational AIcited by 2
  3. 2024Evaluating the Quality of Code Comments Generated by Large Language Models for Novice Programmerscited by 0