# CLOSER-Bench: Evaluating Budgeted Cross-Stage Design Closure for Hardware Agents

arXiv: https://arxiv.org/abs/2607.16632 · HTML: https://mlsanity.com/paper/2607.16632/

## Key facts

- No citing papers in the map yet (data through 7 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 18 Jul 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 11 of 11)

- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [ChatEDA: A Large Language Model Powered Autonomous Agent for EDA](https://mlsanity.com/paper/2308.10204/) (2023)
- [ORFS-agent: Tool-Using Agents for Chip Design Optimization](https://mlsanity.com/paper/2506.08332/) (2025)
- [ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design](https://mlsanity.com/paper/2601.21448/) (2026)
- [HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks](https://mlsanity.com/paper/2604.14709/) (2026)
- [Agentic Hardware Design as Repository-Level Code Evolution](https://mlsanity.com/paper/2606.28279/) (2026)
- [Design Conductor: An agent autonomously builds a 1.5 GHz Linux-capable RISC-V CPU](https://mlsanity.com/paper/2603.08716/) (2026)
- [CktEvo: Repository-Level RTL Code Benchmark for Design Evolution](https://mlsanity.com/paper/2603.08718/) (2026)
- [Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench](https://mlsanity.com/paper/2605.15226/) (2026)
- [HSCO-Bench: An Agent-Driven End-to-End Hardware-Software Co-design Benchmark for Systems-on-Chip](https://mlsanity.com/paper/2605.19399/) (2026)
- [HighTide: An Agent-Curated Open-Source VLSI Benchmark Suite](https://mlsanity.com/paper/2606.04126/) (2026)

## Abstract (arXiv)

Hardware engineering exposes coding agents to a form of long-horizon work that is difficult to capture with pass-at-k: progress is continuous, tool feedback is delayed and heterogeneous, and a backend failure may require revising RTL rather than tuning another physical-design parameter. Existing benchmarks measure RTL generation, repository repair, verification, PPA evolution, or physical implementation, but their different designs and oracles make it hard to determine where an agent succeeds or fails across abstraction boundaries. We introduce CLOSER-Bench, a controlled evaluation protocol for budgeted cross-stage design closure. For one design and one hidden objective, it pairs spec-to-RTL, RTL-to-GDS, and spec-to-GDS tasks, records every simulator, synthesis, STA, and place-and-route invocation, and measures final quality, anytime progress, tool cost, and cross-stage recovery. The benchmark is built on open-source Verilator, Yosys, OpenROAD, KLayout, Sky130, and the Harbor agent harness. A ten-task pilot spanning RTL repair, mutation-based verification, coverage, PPA optimization, design-space exploration, cross-model debugging, and security establishes the executable harness and exposes a sharp completion--closure gap: three agents solve a localized AXI repair task, while the matched verification-closure task separates a frontier agent from two otherwise successful baselines. We further validate a full RTL-to-GDS flow and construct a macro-based AXI/DMA streaming accelerator for the stage-paired evaluation. These results motivate treating hardware closure as a budgeted sequential decision problem rather than a collection of independent code generation tasks.