# Neural Task Synthesis for Visual Programming arXiv: https://arxiv.org/abs/2305.18342 ยท HTML: https://mlsanity.com/paper/2305.18342/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Builds on 28 of its 69 references that are in the map. - First posted on arXiv 26 May 2023. ## Builds on (most cited first, 28 of 28) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021) - [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023) - [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020) - [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021) - [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023) - [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022) - [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022) - [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022) - [MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment](https://mlsanity.com/paper/1709.06298/) (2017) - [DeepCoder: Learning to Write Programs](https://mlsanity.com/paper/1611.01989/) (2016) - [RobustFill: Neural Program Learning under Noisy I/O](https://mlsanity.com/paper/1703.07469/) (2017) - [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022) - [Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis](https://mlsanity.com/paper/1805.04276/) (2018) - [Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models](https://mlsanity.com/paper/2305.15074/) (2023) - [The Synthesizability of Molecules Proposed by Generative Models](https://mlsanity.com/paper/2002.07007/) (2020) - [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022) - [Synthesize, Execute and Debug: Learning to Repair for Neural Program Synthesis](https://mlsanity.com/paper/2007.08095/) (2020) - [Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models](https://mlsanity.com/paper/2302.04662/) (2023) - [Reinforcement Learning for Education: Opportunities and Challenges](https://mlsanity.com/paper/2107.08828/) (2021) - [Synthesizing Tasks for Block-based Programming](https://mlsanity.com/paper/2006.16913/) (2020) - [Zero Shot Learning for Code Education: Rubric Sampling with Deep Learning Inference](https://mlsanity.com/paper/1809.01357/) (2018) - [Synthetic Datasets for Neural Program Synthesis](https://mlsanity.com/paper/1912.12345/) (2019) - [Adaptive Scaffolding in Block-Based Programming via Synthesizing New Tasks as Pop Quizzes](https://mlsanity.com/paper/2303.16359/) (2023) - [From {Solution Synthesis} to {Student Attempt Synthesis} for Block-Based Visual Programming Tasks](https://mlsanity.com/paper/2205.01265/) (2022) ## Built on by (most cited first, 13 of 13) - [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023) - [Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors](https://mlsanity.com/paper/2306.17156/) (2023) - [Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation](https://mlsanity.com/paper/2310.03780/) (2023) - [Evaluating ChatGPT and GPT-4 for Visual Programming](https://mlsanity.com/paper/2308.02522/) (2023) - [Generative AI for Education (GAIED): Advances, Opportunities, and Challenges](https://mlsanity.com/paper/2402.01580/) (2024) - [Large Language Models for In-Context Student Modeling: Synthesizing Student's Behavior in Visual Programming](https://mlsanity.com/paper/2310.10690/) (2023) - [BugSpotter: Automated Generation of Code Debugging Exercises](https://mlsanity.com/paper/2411.14303/) (2024) - [Towards Generalizable Agents in Text-Based Educational Environments: A Study of Integrating RL with LLMs](https://mlsanity.com/paper/2404.18978/) (2024) - [Program Synthesis Benchmark for Visual Programming in XLogoOnline Environment](https://mlsanity.com/paper/2406.11334/) (2024) - [Benchmarking Generative Models on Computational Thinking Tests in Elementary Visual Programming](https://mlsanity.com/paper/2406.09891/) (2024) - [Synthesizing High-Quality Programming Tasks with LLM-based Expert and Student Agents](https://mlsanity.com/paper/2504.07655/) (2025) - [Prompt Programming: A Platform for Dialogue-based Computational Problem Solving with Generative AI Models](https://mlsanity.com/paper/2503.04267/) (2025) - [Visual Set Program Synthesizer](https://mlsanity.com/paper/2603.15997/) (2026) ## Abstract (arXiv) Generative neural models hold great promise in enhancing programming education by synthesizing new content. We seek to design neural models that can automatically generate programming tasks for a given specification in the context of visual programming domains. Despite the recent successes of large generative models like GPT-4, our initial results show that these models are ineffective in synthesizing visual programming tasks and struggle with logical and spatial reasoning. We propose a novel neuro-symbolic technique, NeurTaskSyn, that can synthesize programming tasks for a specification given in the form of desired programming concepts exercised by its solution code and constraints on the visual task. NeurTaskSyn has two components: the first component is trained via imitation learning procedure to generate possible solution codes, and the second component is trained via reinforcement learning procedure to guide an underlying symbolic execution engine that generates visual tasks for these codes. We demonstrate the effectiveness of NeurTaskSyn through an extensive empirical evaluation and a qualitative study on reference tasks taken from the Hour of Code: Classic Maze challenge by Code-dot-org and the Intro to Programming with Karel course by CodeHS-dot-com.