mlsanityData to 7 Oct 2026

QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks

arXiv 2501.17167 ↗First posted 20 Jan 2025cs.SE cs.AIcited by 22

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

We introduce QualityFlow, a dynamic agentic workflow for program synthesis. Given the English description of a programming problem and a set of unit tests, the model's goal is to synthesize the correct program that solves the problem and passes the tests. QualityFlow includes large language model (LLM) agents resembling a software development team, including code generation, testing, and self-debugging. We propose the LLM Quality Checker, which explicitly "imagines" whether the synthesized programs' execution would conform to the unit tests. The Quality Checks dynamically control the workflow, including actions to submit the final answer, clarify the problem statement, and revert previous workflow steps. Our experiments show that the Quality Checker can precisely accept any correct program, mitigate faulty synthesized tests, and prevent potential workflow deviation. QualityFlow establishes the state-of-the-art results on four program synthesis benchmarks: MBPP, HumanEval, and stricter evaluations from MBPP-EvalPlus and HumanEval-EvalPlus.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 33 of its 54 references that are in the map. First posted on arXiv 20 Jan 2025.

Builds on

33 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solving18 May
  2. 2023Self-collaboration Code Generation via ChatGPT15 Apr

Other references · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  11. 2023Teaching Large Language Models to Self-Debugcited by 963
  12. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
19 more
  1. 2021Measuring Coding Challenge Competence With APPScited by 879
  2. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  3. 2022Large Language Models Can Self-Improvecited by 656
  4. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  5. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  6. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  7. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  8. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  9. 2019SPoC: Search-based Pseudocode to Codecited by 176
  10. 2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 134
  11. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 124
  12. 2022Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproductioncited by 110
  13. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110
  14. 2023ChatUniTest: A Framework for LLM-Based Test Generationcited by 94
  15. 2024CYCLE: Learning to Self-Refine the Code Generationcited by 39
  16. 2024A Deep Dive into Large Language Models for Automated Bug Localization and Repaircited by 39
  17. 2023ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarificationcited by 37
  18. 2023INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repaircited by 18
  19. 2022Fix Bugs with Transformer through a Neural-Symbolic Edit Grammarcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 12.

22 papers.

Sort:

By area:

  1. 2026Code as Agent Harnesscited by 40
  2. 2025Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworkscited by 36
  3. 2026From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agentscited by 22
  4. 2025Execution Guided Line-by-Line Code Generationcited by 10
  5. 2025Incoherence as Oracle-less Measure of Error in LLM-Based Code Generationcited by 6
  6. 2026SEMAG: Self-Evolutionary Multi-Agent Code Generationcited by 5
  7. 2025Programming with Pixels: Can Computer-Use Agents do Software Engineering?cited by 4
  8. 2025Nexus: A Lightweight and Scalable Multi-Agent Framework for Complex Tasks Automationcited by 4
  9. 2025LADs: Leveraging LLMs for AI-Driven DevOpscited by 4
  10. 2025Agint: Agentic Graph Compilation for Software Engineering Agentscited by 4
  11. 2025Boosting Open-Source LLMs for Program Repair via Reasoning Transfer and LLM-Guided Reinforcement Learningcited by 2
  12. 2025Risk-Sensitive Agent Compositionscited by 1
10 more
  1. 2026TheoremForge: Scaling up Formal Data Synthesis with Low-Budget Agentic Workflowcited by 1
  2. 2026Team, Then Trim: An Assembly-Line LLM Framework for High-Quality Tabular Data Generationcited by 1
  3. 2026On the Creativity of AI Agentscited by 1
  4. 2026Coding with Eyes: Visual Feedback Unlocks Reliable GUI Code Generating and Debuggingcited by 1
  5. 2026Graph Engineering in the Era of LLM Agents: From Individual Intelligence to System Intelligencecited by 1
  6. 2025LLM-Based Config Synthesis requires Disambiguationcited by 0
  7. 2026ATTest: Agent-Driven Tensor Testing for Deep Learning Library Modulescited by 0
  8. 2026CLEAR: Context Augmentation from Contrastive Learning of Experience via Agentic Reflectioncited by 0
  9. 2026CatalogAgent: A Supervisor-mediated Self-Learning System Enabling Context Engineering for GenAI Modelscited by 0
  10. 2026ACTS-SQL: Agentic and Critic-Oriented Tree-Structured SQL Correctness with Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TDFlow: Agentic Workflows for Test Driven Development
  2. 2026LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation
  3. 2026Consistency Meets Verification: Enhancing Test Generation Quality in Large Language Models Without Ground-Truth Solutions
  4. 2026Diagnosis-Driven Automatic Repair for Agentic Workflow via Symbolic Inference
  5. 2026WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics
  6. 2026AEVAL: From Anecdotal to Deterministic Testing for Agentic Skill Workflows
  7. 2026P3: Joint Program-and-Proof Planning for Verified Code Generation
  8. 2024Planning-Driven Programming: A Large Language Model Programming Workflow