# Generating Planning Feedback for Open-Ended Programming Exercises with LLMs

arXiv: https://arxiv.org/abs/2504.08958 · HTML: https://mlsanity.com/paper/2504.08958/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 53 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Comparing Code Explanations Created by Students and Large Language Models](https://mlsanity.com/paper/2304.03938/) (2023)
- [Fine-tuning ChatGPT for Automatic Scoring](https://mlsanity.com/paper/2310.10072/) (2023)
- [Program Synthesis and Semantic Parsing with Learned Code Idioms](https://mlsanity.com/paper/1906.10816/) (2019)
- [Mining Idioms from Source Code](https://mlsanity.com/paper/1404.0417/) (2014)
- [Matching Exemplar as Next Sentence Prediction (MeNSP): Zero-shot Prompt Learning for Automatic Scoring in Science Education](https://mlsanity.com/paper/2301.08771/) (2023)
- [Exploring the Role of AI Assistants in Computer Science Education: Methods, Implications, and Instructor Perspectives](https://mlsanity.com/paper/2306.03289/) (2023)
- [Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming](https://mlsanity.com/paper/2411.09261/) (2024)
- [Mining Idioms in the Wild](https://mlsanity.com/paper/2107.06402/) (2021)
- [Handwritten Code Recognition for Pen-and-Paper CS Education](https://mlsanity.com/paper/2408.07220/) (2024)

## Built on by (most cited first, 4 of 4)

- [Enhancing Diversity of LLM-Generated Educational Tasks](https://mlsanity.com/paper/2512.23601/) (2025)
- [Beyond Either-Or Reasoning: Transduction and Induction as Cooperative Problem-Solving Paradigms](https://mlsanity.com/paper/2505.14744/) (2025)
- [Designing and Evaluating Chain-of-Hints for Scientific Question Answering](https://mlsanity.com/paper/2510.21087/) (2025)
- [Solver-Aware Decompositions for Programming-by-Example: When Dividing Requires Knowing how to Conquer](https://mlsanity.com/paper/2608.03461/) (2026)

## Abstract (arXiv)

To complete an open-ended programming exercise, students need to both plan a high-level solution and implement it using the appropriate syntax. However, these problems are often autograded on the correctness of the final submission through test cases, and students cannot get feedback on their planning process. Large language models (LLM) may be able to generate this feedback by detecting the overall code structure even for submissions with syntax errors. To this end, we propose an approach that detects which high-level goals and patterns (i.e. programming plans) exist in a student program with LLMs. We show that both the full GPT-4o model and a small variant (GPT-4o-mini) can detect these plans with remarkable accuracy, outperforming baselines inspired by conventional approaches to code analysis. We further show that the smaller, cost-effective variant (GPT-4o-mini) achieves results on par with state-of-the-art (GPT-4o) after fine-tuning, creating promising implications for smaller models for real-time grading. These smaller models can be incorporated into autograders for open-ended code-writing exercises to provide feedback for students' implicit planning skills, even when their program is syntactically incorrect. Furthermore, LLMs may be useful in providing feedback for problems in other domains where students start with a set of high-level solution steps and iteratively compute the output, such as math and physics problems.