# An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc

arXiv: https://arxiv.org/abs/2603.15976 · HTML: https://mlsanity.com/paper/2603.15976/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 28 of its 40 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 28 of 28)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery](https://mlsanity.com/paper/2410.05080/) (2024)
- [SciCode: A Research Coding Benchmark Curated by Scientists](https://mlsanity.com/paper/2407.13168/) (2024)
- [Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks](https://mlsanity.com/paper/2305.10160/) (2023)
- [Establishing Best Practices for Building Rigorous Agentic Benchmarks](https://mlsanity.com/paper/2507.02825/) (2025)
- [Can Large Language Models Write Parallel Code?](https://mlsanity.com/paper/2401.12554/) (2024)
- [Comparing Llama-2 and GPT-3 LLMs for HPC kernels generation](https://mlsanity.com/paper/2309.07103/) (2023)
- [MooseAgent: A LLM Based Multi-agent Framework for Automating Moose Simulation](https://mlsanity.com/paper/2504.08621/) (2025)
- [OMPGPT: A Generative Pre-trained Transformer Model for OpenMP](https://mlsanity.com/paper/2401.16445/) (2024)
- [LLM4VV: Developing LLM-Driven Testsuite for Compiler Validation](https://mlsanity.com/paper/2310.04963/) (2023)
- [ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development](https://mlsanity.com/paper/2602.01655/) (2026)
- [HPC-Coder-V2: Studying Code LLMs Across Low-Resource Parallel Languages](https://mlsanity.com/paper/2412.15178/) (2024)
- [LLM & HPC:Benchmarking DeepSeek's Performance in High-Performance Computing Tasks](https://mlsanity.com/paper/2504.03665/) (2025)
- [FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs](https://mlsanity.com/paper/2512.20732/) (2025)
- [PDE-Agent: A toolchain-augmented multi-agent framework for PDE solving](https://mlsanity.com/paper/2512.16214/) (2025)
- [Do Large Language Models Understand Performance Optimization?](https://mlsanity.com/paper/2503.13772/) (2025)
- [SciML Agents: Write the Solver, Not the Solution](https://mlsanity.com/paper/2509.09936/) (2025)
- [AInsteinBench: Benchmarking Coding Agents on Scientific Repositories](https://mlsanity.com/paper/2512.21373/) (2025)
- [From Legacy Fortran to Portable Kokkos: An Autonomous Agentic AI Workflow](https://mlsanity.com/paper/2509.12443/) (2025)
- [ParEval-Repo: A Benchmark Suite for Evaluating LLMs with Repository-level HPC Translation Tasks](https://mlsanity.com/paper/2506.20938/) (2025)
- [Leveraging LLMs to Automate Energy-Aware Refactoring of Parallel Scientific Codes](https://mlsanity.com/paper/2505.02184/) (2025)
- [Beyond Code Pairs: Dialogue-Based Data Generation for LLM Code Translation](https://mlsanity.com/paper/2512.03086/) (2025)
- [MOSAIC: Multi-agent Orchestration for Task-Intelligent Scientific Coding](https://mlsanity.com/paper/2510.08804/) (2025)
- [AI Assistants to Enhance and Exploit the PETSc Knowledge Base](https://mlsanity.com/paper/2506.20608/) (2025)

## Built on by (most cited first, 1 of 1)

- [Training Numerical Intelligence via Auto-Diagnosis and Skill Discovery](https://mlsanity.com/paper/2610.03872/) (2026)

## Abstract (arXiv)

While LLMs have accelerated scientific code generation, comprehensively evaluating generated code remains challenging. Many benchmarks emphasize functional correctness or task completion, which is insufficient for code built on production HPC libraries, where solver selection, API conventions, memory management, parallel awareness, and performance also matter. We introduce PETSCAgent-Bench, a multidimensional benchmark and agent-based framework for assessing whether AI-generated scientific code uses a production HPC library as an expert would. A tool-augmented evaluator compiles, executes, and measures code and combines deterministic checks with LLM-based assessments in a 14-evaluator pipeline spanning five categories: correctness, performance, code quality, algorithmic appropriateness, and library-specific conventions. A2A and MCP enable black-box evaluation of compatible coding agents. Across realistic PETSc problems, frontier models generate readable, well-structured code but struggle with correctness on challenging problems and with library-specific conventions even when code compiles and runs---limitations that conventional pass/fail evaluation does not capture.