# Enhancing Computer Programming Education with LLMs: A Study on Effective Prompt Engineering for Python Code Generation

arXiv: https://arxiv.org/abs/2407.05437 · HTML: https://mlsanity.com/paper/2407.05437/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 7 Jul 2024.

## Builds on (most cited first, 25 of 25)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [Ask Me Anything: A simple strategy for prompting language models](https://mlsanity.com/paper/2210.02441/) (2022)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://mlsanity.com/paper/2302.07427/) (2023)
- [Is GitHub's Copilot as Bad as Humans at Introducing Vulnerabilities in Code?](https://mlsanity.com/paper/2204.04741/) (2022)
- [ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design](https://mlsanity.com/paper/2303.07839/) (2023)
- [Computing Education in the Era of Generative AI](https://mlsanity.com/paper/2306.02608/) (2023)
- [Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing](https://mlsanity.com/paper/2308.16557/) (2023)
- [Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests](https://mlsanity.com/paper/2306.05715/) (2023)
- [An Empirical Study on the Usage of BERT Models for Code Completion](https://mlsanity.com/paper/2103.07115/) (2021)
- [Testing LLMs on Code Generation with Varying Levels of Prompt Specificity](https://mlsanity.com/paper/2311.07599/) (2023)

## Built on by (most cited first, 9 of 9)

- [EduBench: A Comprehensive Benchmarking Dataset for Evaluating Large Language Models in Diverse Educational Scenarios](https://mlsanity.com/paper/2505.16160/) (2025)
- [The Sociolinguistic Foundations of Language Modeling](https://mlsanity.com/paper/2407.09241/) (2024)
- [The Impact of Prompt Programming on Function-Level Code Generation](https://mlsanity.com/paper/2412.20545/) (2024)
- [Code Generation and Algorithmic Problem Solving Using Llama 3.1 405B](https://mlsanity.com/paper/2409.19027/) (2024)
- [Hyperbolic Large Language Models](https://mlsanity.com/paper/2509.05757/) (2025)
- [Integrating Expert Knowledge into Logical Programs via LLMs](https://mlsanity.com/paper/2502.12275/) (2025)
- [Unlocking the Potential of Past Research: Using Generative AI to Reconstruct Healthcare Simulation Models](https://mlsanity.com/paper/2503.21646/) (2025)
- [Can LLMs Replace Humans During Code Chunking?](https://mlsanity.com/paper/2506.19897/) (2025)
- [PCEval: A Benchmark for Evaluating Physical Computing Capabilities of Large Language Models](https://mlsanity.com/paper/2601.02404/) (2025)

## Abstract (arXiv)

Large language models (LLMs) and prompt engineering hold significant potential for advancing computer programming education through personalized instruction. This paper explores this potential by investigating three critical research questions: the systematic categorization of prompt engineering strategies tailored to diverse educational needs, the empowerment of LLMs to solve complex problems beyond their inherent capabilities, and the establishment of a robust framework for evaluating and implementing these strategies. Our methodology involves categorizing programming questions based on educational requirements, applying various prompt engineering strategies, and assessing the effectiveness of LLM-generated responses. Experiments with GPT-4, GPT-4o, Llama3-8b, and Mixtral-8x7b models on datasets such as LeetCode and USACO reveal that GPT-4o consistently outperforms others, particularly with the "multi-step" prompt strategy. The results show that tailored prompt strategies significantly enhance LLM performance, with specific strategies recommended for foundational learning, competition preparation, and advanced problem-solving. This study underscores the crucial role of prompt engineering in maximizing the educational benefits of LLMs. By systematically categorizing and testing these strategies, we provide a comprehensive framework for both educators and students to optimize LLM-based learning experiences. Future research should focus on refining these strategies and addressing current LLM limitations to further enhance educational outcomes in computer programming instruction.