mlsanityData to 14 Oct 2026

Can Coding Agents Reproduce Official Statistics? Metadata, Retry Budget and the Limits of Execution Feedback in a Controlled Eurostat Benchmark

arXiv 2609.22222 ↗First posted 2 Sep 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Large language models can generate executable data-analysis code, but successful execution is not equivalent to a valid official-statistics result. This study asks whether authoritative metadata and execution feedback improve the reproducibility of Eurostat answers produced by a coding agent, and isolates what execution feedback actually contributes. A benchmark of 30 natural-language tasks covering seven domains, seven Eurostat datasets and four difficulty tiers was run under four conditions: task only (A), task plus a frozen dataset metadata card (B), metadata plus a repair loop driven by sanitized execution feedback (C), and metadata plus the same attempt budget with no diagnostics of any kind (D). Claude Sonnet 5 generated Python through the Anthropic Messages API in three independent replicates, yielding 360 task-runs. Exact correctness required successful execution, the correct dataset, filters, output shape, values and unit. A companion experiment run under an under-specified output contract, in which the required ranking key and unit representation were never stated to the model, understated condition C by 23.4 points, showing that evaluator and contract design can dominate measured agent error. Reliable statistical coding agents need semantic validation against frozen specifications, a fully specified output contract, and a retry budget - not execution diagnostics.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 5 of its 12 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

5 of 12 references.

References · 5

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  3. 2023Teaching Large Language Models to Self-Debugcited by 988
  4. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  5. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches
  2. 2026AI Coding Agents Can Reproduce Social Science Findings
  3. 2026Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase
  4. 2026Re-executing Coding-Agent Evaluations: Unchanged Decisions and Unresolved Policy Risk
  5. 2026Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks
  6. 2026Skill-based Agentic Evaluation for Real-time Data Science Tasks
  7. 2026Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results
  8. 2026Beyond the Text: Verifying That Agent-Written Papers Are Backed by Their Artifacts