mlsanityData to 14 Oct 2026

Synthesizing High-Quality Programming Tasks with LLM-based Expert and Student Agents

arXiv 2504.07655 ↗First posted 10 Apr 2025cs.AI cs.CYcited by 3

Not on the map. Why?

Abstract (arXiv)

Generative AI is transforming computing education by enabling the automatic generation of personalized content and feedback. We investigate its capabilities in providing high-quality programming tasks to students. Despite promising advancements in task generation, a quality gap remains between AI-generated and expert-created tasks. The AI-generated tasks may not align with target programming concepts, could be incomprehensible to students, or may contain critical issues such as incorrect tests. Existing works often require interventions from human teachers for validation. We address these challenges by introducing PyTaskSyn, a novel synthesis technique that first generates a programming task and then decides whether it meets certain quality criteria to be given to students. The key idea is to break this process into multiple stages performed by expert and student agents simulated using both strong and weaker generative models. Through extensive evaluation, we show that PyTaskSyn significantly improves task quality compared to baseline techniques and showcases the importance of each specialized agent type in our validation pipeline. Additionally, we conducted user studies using our publicly available web application and show that PyTaskSyn can deliver high-quality programming tasks comparable to expert-designed ones while reducing workload and costs, and being more engaging than programming tasks that are available in online resources.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 10 Apr 2025.

Builds on

32 of 42 references.

References · 32

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  5. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  6. 2023Teaching Large Language Models to Self-Debugcited by 988
  7. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
  8. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
  9. 2024Simulating Classroom Education with LLM-Empowered Agentscited by 127
  10. 2022Automatic Generation of Programming Exercises and Code Explanations using Large Language Modelscited by 126
  11. 2022Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programmingcited by 91
  12. 2023The Robots are Here: Navigating the Generative AI Revolution in Computing Educationcited by 86
20 more
  1. 2022Using Large Language Models to Enhance Programming Error Messagescited by 67
  2. 2024Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluationcited by 58
  3. 2023Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutorscited by 39
  4. 2023Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Educationcited by 39
  5. 2024MathVC: An LLM-Simulated Multi-Character Virtual Classroom for Mathematics Educationcited by 37
  6. 2023Generating High-Precision Feedback for Programming Syntax Errors using Large Language Modelscited by 34
  7. 2024Knowledge Graphs as Context Sources for LLM-Based Explanations of Learning Recommendationscited by 31
  8. 2023How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debuggingcited by 30
  9. 2023Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validationcited by 29
  10. 2020Synthesizing Tasks for Block-based Programmingcited by 19
  11. 2023Harnessing LLMs in Curricular Design: Using GPT-4 to Support Authoring of Learning Objectivescited by 16
  12. 2023Neural Task Synthesis for Visual Programmingcited by 13
  13. 2024Generative AI for Education (GAIED): Advances, Opportunities, and Challengescited by 13
  14. 2024Evaluating Contextually Personalized Programming Exercises Created with Generative AIcited by 13
  15. 2023Ruffle&Riley: Towards the Automated Induction of Conversational Tutoring Systemscited by 12
  16. 2022Toward Trustworthy Neural Program Synthesiscited by 11
  17. 2023Large Language Models for In-Context Student Modeling: Synthesizing Student's Behavior in Visual Programmingcited by 8
  18. 2024Automating Personalized Parsons Problems with Customized Contexts and Conceptscited by 4
  19. 2023The Behavior of Large Language Models When Prompted to Generate Code Explanationscited by 3
  20. 2024Task Synthesis for Elementary Visual Programming in XLogoOnline Environmentcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Enhancing Diversity of LLM-Generated Educational Taskscited by 2
  2. 2026Teaching Language Models How to Code Like Learners: Conversational Serialization for Student Simulationcited by 1
  3. 2026AgentCAT: Simulating Computerized Adaptive Testing via Multi-Agent Large Language Modelscited by 0