mlsanityData to 14 Oct 2026

Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback

arXiv 2506.18315 ↗First posted 23 Jun 2025cs.SE cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

LLMs excel at code generation, yet ensuring the functional correctness of their outputs remains a persistent challenge. While recent studies have applied Test-Driven Development (TDD) to refine code, these methods are often undermined by poor feedback quality, stemming from the scarcity of high-quality test cases and noisy signals from auto-generated ones. In this work, we shift the focus from test quantity to feedback quality. We introduce the Property-Generated Solver (PGS), a novel paradigm designed to generate highly effective feedback via two principles: it must be property-oriented, to provide semantic guidance beyond simple I/O mismatches, and structurally minimal, to reduce cognitive load and isolate root causes. PGS operates by checking high-level program properties (e.g., a sorting function must produce a non-decreasing sequence) then providing the simplest failing counterexample to the LLM. By adhering to these principles, this targeted feedback mechanism leads to significant performance gains. Specifically, PGS achieves an improvement of up to 13.4% in pass@1 against other TDD-based methods and an over 64% fix rate on problems where the model initially failed. This property-driven, minimal feedback steers LLMs toward correct and generalizable solutions. Across diverse benchmarks, PGS demonstrates superior performance, achieving a bug fix rate 1.4x-1.6x higher than the strongest debugging-based approaches and establishing a new state-of-the-art in automated code refinement.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

29 of 40 references.

References · 29

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  11. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  12. 2024Qwen2.5-Coder Technical Reportcited by 1,203
17 more
  1. 2023Teaching Large Language Models to Self-Debugcited by 988
  2. 2022CodeT: Code Generation with Generated Testscited by 384
  3. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  4. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  5. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 239
  6. 2024Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-stepcited by 127
  7. 2025Competitive Programming with Large Reasoning Modelscited by 119
  8. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 111
  9. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 86
  10. 2021Break-It-Fix-It: Unsupervised Learning for Program Repaircited by 71
  11. 2023Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?cited by 57
  12. 2024From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debuggingcited by 46
  13. 2023SelfEvolve: A Code Evolution Framework via Large Language Modelscited by 43
  14. 2023Conversational Automated Program Repaircited by 41
  15. 2023Can Large Language Models Write Good Property-Based Tests?cited by 31
  16. 2024A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinementcited by 21
  17. 2022Toward Trustworthy Neural Program Synthesiscited by 11

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystemcited by 8
  2. 2025NALA_MAINZ at BLP-2025 Task 2: A Multi-agent Approach for Bangla Instruction to Python Code Generationcited by 0