# Automatic Harness Evolution for Hardware Design Verification: Can LLMs Consolidate Gains Across Discovered Harnesses?

arXiv: https://arxiv.org/abs/2609.28908 · HTML: https://mlsanity.com/paper/2609.28908/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 20 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 15 of 15)

- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://mlsanity.com/paper/2507.19457/) (2025)
- [AFlow: Automating Agentic Workflow Generation](https://mlsanity.com/paper/2410.10762/) (2024)
- [Automated Design of Agentic Systems](https://mlsanity.com/paper/2408.08435/) (2024)
- [Meta-Harness: End-to-End Optimization of Model Harnesses](https://mlsanity.com/paper/2603.28052/) (2026)
- [ChipNeMo: Domain-Adapted LLMs for Chip Design](https://mlsanity.com/paper/2311.00176/) (2023)
- [Self-Harness: Harnesses That Improve Themselves](https://mlsanity.com/paper/2606.09498/) (2026)
- [Rethinking the Evaluation of Harness Evolution for Agents](https://mlsanity.com/paper/2607.12227/) (2026)
- [Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation](https://mlsanity.com/paper/2408.11053/) (2024)
- [Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification](https://mlsanity.com/paper/2506.14074/) (2025)
- [Stop Comparing LLM Agents Without Disclosing the Harness](https://mlsanity.com/paper/2605.23950/) (2026)
- [Recursive Harness Self-Improvement](https://mlsanity.com/paper/2607.15524/) (2026)
- [HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?](https://mlsanity.com/paper/2609.01437/) (2026)
- [Agentic Hardware Design as Repository-Level Code Evolution](https://mlsanity.com/paper/2606.28279/) (2026)
- [EvolveNet: Collaborative Harness Evolution for Agent Self-Improvement](https://mlsanity.com/paper/2608.04968/) (2026)

## Built on by (most cited first, 1 of 1)

- [What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects](https://mlsanity.com/paper/2610.05533/) (2026)

## Abstract (arXiv)

Agent behavior depends on the harness surrounding a language model, but it remains unclear whether language models can reliably improve such harnesses for hardware-design tasks. We study automatic harness evolution around a fixed subject model on 12 proprietary design-verification root-cause localization tasks. Across five trials per task, automatically evolved harnesses increased completed attempts by 71-76% and any-hit task coverage by 80-100%, while total correct attempts improved by only 18-24%. The strongest success reproducible at least twice result improved by one task, and later candidates exchanged gains across tasks rather than preserving them. An auxiliary candidate improved on a four-task validation set excluded from search but tied its baseline on a subsequent 12-task replay containing both search and validation tasks, so the selected gain did not persist across the full pool. Across the tested lineage, useful search, evidence, and finalization behaviors appeared in different candidates but did not consistently consolidate into a single harness that dominated across tasks and metrics. In a separate CVDP cross-benchmark case study, an automatically evolved defined-width repair harness produced 35.6% more functional passes than its 142-task reference baseline; the final functional verifier scored completed outputs but was not shown to the subject agent during repair. These results support archive-aware selection when evolution yields complementary specializations without consistent consolidation.