mlsanityData to 14 Oct 2026

Agentic Synthesis against Counterexample-Supplemented Sketches

arXiv 2607.15854 ↗First posted 17 Jul 2026cs.SE cs.AIcited by 0

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Coding agents can fix a failing example without preserving the domain rule that made it fail. We present agentic synthesis against counterexample-supplemented sketches, a repository-native method for systems whose policy is discovered during implementation. A human starts with a partial sketch, and a coding agent compiles a replaceable projection. When simulation exposes missing or mistaken policy, an operator approves the corrected behavior and the minimum general rule the case authorizes. Every Developer call names its change authority and the rules, holes, anchors, and approved behavior that must survive. Conflict or ambiguous permission leaves the files unchanged and produces a clarification question. A complete archive preserves provenance; a curated regression set gates distinct boundaries. Before another candidate is revealed, the active case and curated regressions must pass both deterministic approved-output comparison and a separate review against the current sketch. Periodic clean regeneration tests whether the sketch carries the learned policy. We demonstrate the method with CatSynth, a captured synthetic application. In one open-world run with GPT-5.4-mini, 8 of 14 frozen candidates became counterexamples. Under the corrected protocol, replay-all, evolved-sketch rebuild, and retained Sketch-CE each passed all 8 accepted cases. They passed 14, 17, and 16 of 21 withheld cases, respectively. Sketch review rejected premature empty-input and tag policies and restored dropped anchors; adjudicated reviewer errors did not become policy. One model and one reveal order cannot establish general correctness or superiority. On this suite, the second check exposed drift hidden by deterministic replay, and the reviewed sketch passed three more withheld cases than raw example replay.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 5 of its 28 references that are in the map. First posted on arXiv 17 Jul 2026.

Builds on

5 of 28 references.

References · 5

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  3. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  4. 2025Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generationcited by 5
  5. 2024Natural Language-Oriented Programming (NLOP): Towards Democratizing Software Creationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Teaching Agents to Code Reliably
  2. 2026Rebuild Dossier: Mechanically-Enforced Specs for Agentic App Rebuilds, and What Model-Tier Failures Reveal
  3. 2026SEVerA: Verified Synthesis of Self-Evolving Agents
  4. 2026ExecCritic: Learn to Test, Test to Improve for Coding Agents
  5. 2026Independent Patch Verification for Coding Agents with a Bidirectional Reconstruct-and-Verify Framework
  6. 2026Humanize: Judgement Engineering for Agentic Coding
  7. 2026Inductive Deductive Synthesis: Enabling AI to Generate Formally Verified Systems
  8. 2026ADIAS: Automated Design of Interactive Agentic Systems