# LLMLOOP: Improving LLM-Generated Code and Tests through Automated Iterative Feedback Loops

arXiv: https://arxiv.org/abs/2603.23613 · HTML: https://mlsanity.com/paper/2603.23613/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 24 Mar 2026.

## Builds on (most cited first, 12 of 12)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](https://mlsanity.com/paper/2303.17568/) (2023)
- [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023)
- [Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction](https://mlsanity.com/paper/2209.11515/) (2022)
- [Pynguin: Automated Unit Test Generation for Python](https://mlsanity.com/paper/2202.05218/) (2022)
- [The Future of AI-Driven Software Engineering](https://mlsanity.com/paper/2406.07737/) (2024)
- [LLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities](https://mlsanity.com/paper/2309.13574/) (2023)
- [Do LLMs generate test oracles that capture the actual or the expected program behaviour?](https://mlsanity.com/paper/2410.21136/) (2024)
- [Comparing large language models and human programmers for generating programming code](https://mlsanity.com/paper/2403.00894/) (2024)
- [Automatic Code Generation using Pre-Trained Language Models](https://mlsanity.com/paper/2102.10535/) (2021)
- [An Empirical Study on Self-correcting Large Language Models for Data Science Code Generation](https://mlsanity.com/paper/2408.15658/) (2024)

## Built on by (most cited first, 8 of 8)

- [Beyond Fail-to-Pass: Iterative Hardening of Co-Generated Bug Reproduction Tests and Fixes](https://mlsanity.com/paper/2607.19843/) (2026)
- [ExecCritic: Learn to Test, Test to Improve for Coding Agents](https://mlsanity.com/paper/2609.09133/) (2026)
- [VeriSimpl: Robust Optimization Modeling from Natural Language using Simplification-based Verification](https://mlsanity.com/paper/2607.20474/) (2026)
- [MR-Coupler: Automated Metamorphic Test Generation via Functional Coupling Analysis](https://mlsanity.com/paper/2604.10126/) (2026)
- [Natural-Language to SysMLv2 Translation via Conformance-Driven Iterative Refinement](https://mlsanity.com/paper/2607.14162/) (2026)
- [Comprendia: AI-Augmented Code Comprehension](https://mlsanity.com/paper/2608.10290/) (2026)
- [Segment-Level Agentic Topic Modeling for Improved Data Exploration and Resource Efficiency](https://mlsanity.com/paper/2609.31460/) (2026)
- [ReMCTS: Reflection-Enhanced Monte Carlo Tree Search for Code Generation](https://mlsanity.com/paper/2609.34717/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are showing remarkable performance in generating source code, yet the generated code often has issues like compilation errors or incorrect code. Researchers and developers often face wasted effort in implementing checks and refining LLM-generated code, frequently duplicating their efforts. This paper presents LLMLOOP, a framework that automates the refinement of both source code and test cases produced by LLMs. LLMLOOP employs five iterative loops: resolving compilation errors, addressing static analysis issues, fixing test case failures, and improving test quality through mutation analysis. These loops ensure the generation of high-quality test cases that serve as both a validation mechanism and a regression test suite for the generated code. We evaluated LLMLOOP on HUMANEVAL-X, a recent benchmark of programming tasks. Results demonstrate the tool's effectiveness in refining LLM-generated outputs.