# Revisit Self-Debugging with Self-Generated Tests for Code Generation arXiv: https://arxiv.org/abs/2501.12793 ยท HTML: https://mlsanity.com/paper/2501.12793/ ## Key facts - Cited by 22 papers in the map (data through 14 Oct 2026). - Placed in Code and instruction data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs. - Builds on 16 of its 31 references that are in the map. - First posted on arXiv 22 Jan 2025. ## Placed in - Synthetic data and self-improvement for LLMs / [Code and instruction data synthesis](https://mlsanity.com/synth/code-and-instruction-data-synthesis/) ## Builds on (most cited first, 16 of 16) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021) - [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024) - [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024) - [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024) - [Self-collaboration Code Generation via ChatGPT](https://mlsanity.com/paper/2304.07590/) (2023) - [Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering](https://mlsanity.com/paper/2401.08500/) (2024) - [A Survey on Self-Evolution of Large Language Models](https://mlsanity.com/paper/2404.14387/) (2024) - [Reasoning Runtime Behavior of a Program with LLM: How Far Are We?](https://mlsanity.com/paper/2403.16437/) (2024) - [SelfEvolve: A Code Evolution Framework via Large Language Models](https://mlsanity.com/paper/2306.02907/) (2023) - [LeDex: Training LLMs to Better Self-Debug and Explain Code](https://mlsanity.com/paper/2405.18649/) (2024) - [AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation](https://mlsanity.com/paper/2312.13010/) (2023) - [TraceFixer: Execution Trace-Driven Program Repair](https://mlsanity.com/paper/2304.12743/) (2023) ## Built on by (most cited first, 22 of 22) - [VisCoder: Fine-Tuning LLMs for Executable Python Visualization Code Generation](https://mlsanity.com/paper/2506.03930/) (2025) - [RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generation](https://mlsanity.com/paper/2502.09183/) (2025) - [InfCode: Adversarial Iterative Refinement of Tests and Patches for Reliable Software Issue Resolution](https://mlsanity.com/paper/2511.16004/) (2025) - [The Debugging Decay Index: Rethinking Debugging Strategies for Code LLMs](https://mlsanity.com/paper/2506.18403/) (2025) - [How Many Tries Does It Take? Iterative Self-Repair in LLM Code Generation Across Model Scales and Benchmarks](https://mlsanity.com/paper/2604.10508/) (2026) - [Code Reasoning for Software Engineering Tasks: A Survey and A Call to Action](https://mlsanity.com/paper/2506.13932/) (2025) - [REAgent: Requirement-Driven LLM Agents for Software Issue Resolution](https://mlsanity.com/paper/2604.06861/) (2026) - [Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents](https://mlsanity.com/paper/2607.24300/) (2026) - [InfCode-C++: Intent-Guided Semantic Retrieval and AST-Structured Search for C++ Issue Resolution](https://mlsanity.com/paper/2511.16005/) (2025) - [ExecCritic: Learn to Test, Test to Improve for Coding Agents](https://mlsanity.com/paper/2609.09133/) (2026) - [CP-SynC: Multi-Agent Zero-Shot Constraint Modeling in MiniZinc with Synthesized Checkers](https://mlsanity.com/paper/2605.01675/) (2026) - [Selection Without Signal, Recovery Through Expression: A Measurement Study of Post-Hoc Falsification Operators for Frozen Small Code Models](https://mlsanity.com/paper/2606.16999/) (2026) - [Complementarity-driven Representation Learning for Multi-modal Knowledge Graph Completion](https://mlsanity.com/paper/2507.20620/) (2025) - [Self-Improving Code Generation via Semantic Entropy and Behavioral Consensus](https://mlsanity.com/paper/2603.29292/) (2026) - [Agentic MIP Research: Accelerated Constraint Handler Generation](https://mlsanity.com/paper/2605.09186/) (2026) - [Mutual Evidential Deep Learning for Medical Image Segmentation](https://mlsanity.com/paper/2505.12418/) (2025) - [Co-Evidential Fusion with Information Volume for Medical Image Segmentation](https://mlsanity.com/paper/2506.02492/) (2025) - [Adaptive Fuzzy Time Series Forecasting via Partially Asymmetric Convolution and Sub-Sliding Window Fusion](https://mlsanity.com/paper/2507.20641/) (2025) - [Inference-Time Code Selection via Symbolic Equivalence Partitioning](https://mlsanity.com/paper/2604.06485/) (2026) - [COMPAS: Difficulty-Aware Joint Search for Optimizing Code Generation](https://mlsanity.com/paper/2608.04336/) (2026) - [ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generation](https://mlsanity.com/paper/2608.04439/) (2026) - [Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation](https://mlsanity.com/paper/2610.15797/) (2026) ## Abstract (arXiv) Large language models (LLMs) have shown significant advancements in code generation, but still face challenges on tasks beyond their basic capabilities. Recently, the notion of self-debugging has been proposed to boost the performance of code generation by leveraging execution feedback from tests. Despite its promise, the availability of high-quality tests in real-world scenarios is limited. In this context, self-debugging with self-generated tests is a promising solution but lacks a full exploration of its limitations and practical potential. Therefore, we investigate its efficacy on diverse programming problems. To deepen our understanding, we propose two distinct paradigms for the process: post-execution and in-execution self-debugging. Within the scope of self-contained Python programming tasks, we find that post-execution self-debugging struggles on basic problems but shows potential for improvement on competitive ones, due to the bias introduced by self-generated tests. On the other hand, in-execution self-debugging enables LLMs to mitigate the bias by solely leveraging intermediate states during execution, thereby enhancing code generation.