# QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks

arXiv: https://arxiv.org/abs/2501.17167 · HTML: https://mlsanity.com/paper/2501.17167/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 33 of its 54 references that are in the map.
- First posted on arXiv 20 Jan 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 33)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://mlsanity.com/paper/2205.12548/) (2022)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [Debating with More Persuasive LLMs Leads to More Truthful Answers](https://mlsanity.com/paper/2402.06782/) (2024)
- [Self-collaboration Code Generation via ChatGPT](https://mlsanity.com/paper/2304.07590/) (2023)
- [MapCoder: Multi-Agent Code Generation for Competitive Problem Solving](https://mlsanity.com/paper/2405.11403/) (2024)
- [SPoC: Search-based Pseudocode to Code](https://mlsanity.com/paper/1906.04908/) (2019)
- [An Empirical Study of the Non-determinism of ChatGPT in Code Generation](https://mlsanity.com/paper/2308.02828/) (2023)
- [CodeGen2: Lessons for Training LLMs on Programming and Natural Languages](https://mlsanity.com/paper/2305.02309/) (2023)
- [Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction](https://mlsanity.com/paper/2209.11515/) (2022)
- [Self-Edit: Fault-Aware Code Editor for Code Generation](https://mlsanity.com/paper/2305.04087/) (2023)
- [ChatUniTest: A Framework for LLM-Based Test Generation](https://mlsanity.com/paper/2305.04764/) (2023)
- [CYCLE: Learning to Self-Refine the Code Generation](https://mlsanity.com/paper/2403.18746/) (2024)
- [A Deep Dive into Large Language Models for Automated Bug Localization and Repair](https://mlsanity.com/paper/2404.11595/) (2024)

## Built on by (most cited first, 22 of 22)

- [Code as Agent Harness](https://mlsanity.com/paper/2605.18747/) (2026)
- [Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks](https://mlsanity.com/paper/2505.07473/) (2025)
- [From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agents](https://mlsanity.com/paper/2603.22386/) (2026)
- [Execution Guided Line-by-Line Code Generation](https://mlsanity.com/paper/2506.10948/) (2025)
- [Incoherence as Oracle-less Measure of Error in LLM-Based Code Generation](https://mlsanity.com/paper/2507.00057/) (2025)
- [Nexus: A Lightweight and Scalable Multi-Agent Framework for Complex Tasks Automation](https://mlsanity.com/paper/2502.19091/) (2025)
- [SEMAG: Self-Evolutionary Multi-Agent Code Generation](https://mlsanity.com/paper/2603.15707/) (2026)
- [Programming with Pixels: Can Computer-Use Agents do Software Engineering?](https://mlsanity.com/paper/2502.18525/) (2025)
- [LADs: Leveraging LLMs for AI-Driven DevOps](https://mlsanity.com/paper/2502.20825/) (2025)
- [Agint: Agentic Graph Compilation for Software Engineering Agents](https://mlsanity.com/paper/2511.19635/) (2025)
- [Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligence](https://mlsanity.com/paper/2608.21156/) (2026)
- [Boosting Open-Source LLMs for Program Repair via Reasoning Transfer and LLM-Guided Reinforcement Learning](https://mlsanity.com/paper/2506.03921/) (2025)
- [Risk-Sensitive Agent Compositions](https://mlsanity.com/paper/2506.04632/) (2025)
- [TheoremForge: Scaling up Formal Data Synthesis with Low-Budget Agentic Workflow](https://mlsanity.com/paper/2601.17332/) (2026)
- [Team, Then Trim: An Assembly-Line LLM Framework for High-Quality Tabular Data Generation](https://mlsanity.com/paper/2602.04785/) (2026)
- [On the Creativity of AI Agents](https://mlsanity.com/paper/2604.13242/) (2026)
- [Coding with Eyes: Visual Feedback Unlocks Reliable GUI Code Generating and Debugging](https://mlsanity.com/paper/2604.19750/) (2026)
- [LLM-Based Config Synthesis requires Disambiguation](https://mlsanity.com/paper/2507.12443/) (2025)
- [ATTest: Agent-Driven Tensor Testing for Deep Learning Library Modules](https://mlsanity.com/paper/2602.13987/) (2026)
- [CLEAR: Context Augmentation from Contrastive Learning of Experience via Agentic Reflection](https://mlsanity.com/paper/2604.07487/) (2026)
- [CatalogAgent: A Supervisor-mediated Self-Learning System Enabling Context Engineering for GenAI Models](https://mlsanity.com/paper/2607.14396/) (2026)
- [ACTS-SQL: Agentic and Critic-Oriented Tree-Structured SQL Correctness with Large Language Models](https://mlsanity.com/paper/2608.15145/) (2026)

## Abstract (arXiv)

We introduce QualityFlow, a dynamic agentic workflow for program synthesis. Given the English description of a programming problem and a set of unit tests, the model's goal is to synthesize the correct program that solves the problem and passes the tests. QualityFlow includes large language model (LLM) agents resembling a software development team, including code generation, testing, and self-debugging. We propose the LLM Quality Checker, which explicitly "imagines" whether the synthesized programs' execution would conform to the unit tests. The Quality Checks dynamically control the workflow, including actions to submit the final answer, clarify the problem statement, and revert previous workflow steps. Our experiments show that the Quality Checker can precisely accept any correct program, mitigate faulty synthesized tests, and prevent potential workflow deviation. QualityFlow establishes the state-of-the-art results on four program synthesis benchmarks: MBPP, HumanEval, and stricter evaluations from MBPP-EvalPlus and HumanEval-EvalPlus.