mlsanityData to 7 Oct 2026

BUILD-AND-FIND: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases

arXiv 2605.06136 ↗First posted 7 May 2026cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Most coding-agent benchmarks ask whether generated code behaves correctly. That remains essential, but repository-level engineering is increasingly agent-managed: one agent writes a repository, and later agents inspect, audit, or extend it as working context. In that setting, a generated repository is not only an answer to a task but also a communication artifact for future work. Even when strong agents nearly satisfy the visible behavioral objective, repositories can differ in how clearly they expose the intended behavior and design choices behind that behavior. We introduce BUILD-AND-FIND, a protocol for evaluating whether downstream agents can recover those intended choices from generated repositories, and how much inspection that recovery requires. For each task, a builder sees a hidden repository specification and creates a codebase; a finder sees only the codebase and a specification-traced multiple-choice question bank. The protocol separates behavioral correctness from artifact-side recovery and reports recovery accuracy, repeatability, implementation coverage, and inspection effort. Accuracy and stability act as gates: effort is interpreted only when recovery succeeds reliably. Among artifacts from which the same intent can be recovered, lower effort by the same finder suggests that the artifact makes that intent easier to locate. Question-only and spec-only controls quantify generic priors and specification access, while audits separate omitted claims from finder failures and check whether correct answers cite artifact evidence. In the released high-prior task pack, recovery accuracy is near saturation, so inspection effort and finder-specific effects provide the main panel-local comparison.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

18 of 27 references.

References · 18

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  6. 2021Measuring Coding Challenge Competence With APPScited by 879
  7. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  8. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
  9. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  10. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  11. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  12. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
6 more
  1. 2024Croissant: A Metadata Format for ML-Ready Datasetscited by 79
  2. 2023Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issuescited by 78
  3. 2025Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineeringcited by 64
  4. 2025SWE-QA: Can Language Models Answer Repository-level Code Questions?cited by 37
  5. 2024InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Modelscited by 13
  6. 2021Evaluating Code Readability and Legibility: An Examination of Human-centric Studiescited by 8

Built on by

Not cited yet.