# Benchmarking LLMs for Verilog Design Flows

arXiv: https://arxiv.org/abs/2607.22759 · HTML: https://mlsanity.com/paper/2607.22759/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 17 references that are in the map.
- First posted on arXiv 23 Jul 2026.

## Builds on (most cited first, 12 of 12)

- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [VeriGen: A Large Language Model for Verilog Code Generation](https://mlsanity.com/paper/2308.00708/) (2023)
- [Benchmarking Large Language Models for Automated Verilog RTL Code Generation](https://mlsanity.com/paper/2212.11140/) (2022)
- [Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback](https://mlsanity.com/paper/2403.16792/) (2024)
- [AutoBench: Automatic Testbench Generation and Evaluation Using LLMs for HDL Design](https://mlsanity.com/paper/2407.03891/) (2024)
- [AutoVCoder: A Systematic Framework for Automated Verilog Code Generation using LLMs](https://mlsanity.com/paper/2407.18333/) (2024)
- [HaVen: Hallucination-Mitigated LLM for Verilog Code Generation Aligned with HDL Engineers](https://mlsanity.com/paper/2501.04908/) (2025)
- [VeriThoughts: Enabling Automated Verilog Code Generation using Reasoning and Formal Verification](https://mlsanity.com/paper/2505.20302/) (2025)
- [ReasoningV: Efficient Verilog Code Generation with Adaptive Hybrid Reasoning Model](https://mlsanity.com/paper/2504.14560/) (2025)
- [VeriDebug: A Unified LLM for Verilog Debugging via Contrastive Embedding and Guided Correction](https://mlsanity.com/paper/2504.19099/) (2025)
- [MetRex: A Benchmark for Verilog Code Metric Reasoning Using LLMs](https://mlsanity.com/paper/2411.03471/) (2024)
- [LLM-VeriPPA: Power, Performance, and Area Optimization aware Verilog Code Generation with Large Language Models](https://mlsanity.com/paper/2510.15899/) (2025)

## Abstract (arXiv)

Large language models (LLMs) show promise in code generation, but their capabilities to produce correct, synthesizable hardware description language (HDL) code still remain to be properly benchmarked. Existing evaluations are primarily relying on pass@k metrics and lack proper end-to-end toolchain validation. This paper presents a reproducible benchmarking platform that evaluates open-source LLMs on Verilog RTL generation across 50 curated tasks consisting of combinational, sequential, finite state machine (FSM), and mixed designs. The pipeline consisting of constrained prompting, post-processing, and semantic-aware iterative refinement with waveform analysis, formal equivalence verification, and Abstract Syntax Tree (AST)-based repair validates the generated code via Verilator compilation and Icarus Verilog simulation. Across the 12 benchmarks and the 1,610 total runs evaluating three models of different sizes (Llama-3-8B, StarCoder2-7B, and TinyLlama-1.1B), the pipeline improved syntax validity from 0% to a 70.43% average and simulation pass rate to 51.8% across three open-source models. Most notably TinyLlama (1.1B parameters) achieved the highest individual syntax validity at 80.0%, with functional correctness comparable to the 8B model. The platform and dataset are open-source, enabling reproducible evaluation of generative AI for hardware design workflows.