# Evaluating LLMs for Hardware Design and Test

arXiv: https://arxiv.org/abs/2405.02326 · HTML: https://mlsanity.com/paper/2405.02326/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 26 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [VeriGen: A Large Language Model for Verilog Code Generation](https://mlsanity.com/paper/2308.00708/) (2023)
- [ChipNeMo: Domain-Adapted LLMs for Chip Design](https://mlsanity.com/paper/2311.00176/) (2023)
- [Chip-Chat: Challenges and Opportunities in Conversational Hardware Design](https://mlsanity.com/paper/2305.13243/) (2023)
- [Benchmarking Large Language Models for Automated Verilog RTL Code Generation](https://mlsanity.com/paper/2212.11140/) (2022)
- [DAVE: Deriving Automatically Verilog from English](https://mlsanity.com/paper/2009.01026/) (2020)
- [(Security) Assertions by Large Language Models](https://mlsanity.com/paper/2306.14027/) (2023)
- [FLAG: Finding Line Anomalies (in code) with Generative AI](https://mlsanity.com/paper/2306.12643/) (2023)

## Built on by (most cited first, 8 of 8)

- [Automatically Improving LLM-based Verilog Generation using EDA Tool Feedback](https://mlsanity.com/paper/2411.11856/) (2024)
- [Can Reasoning Models Reason about Hardware? An Agentic HLS Perspective](https://mlsanity.com/paper/2503.12721/) (2025)
- [Large Language Model for Verilog Code Generation: Literature Review and the Road Ahead](https://mlsanity.com/paper/2512.00020/) (2025)
- [Large Language Models (LLMs) for Electronic Design Automation (EDA)](https://mlsanity.com/paper/2508.20030/) (2025)
- [Deep Representation Learning for Electronic Design Automation](https://mlsanity.com/paper/2505.02105/) (2025)
- [Controlling Context: Generative AI at Work in Integrated Circuit Design and Other High-Precision Domains](https://mlsanity.com/paper/2506.14567/) (2025)
- [OpenRTLSet: A Fully Open-Source Dataset for Large Language Model-based Verilog Module Design](https://mlsanity.com/paper/2606.10285/) (2026)
- [Agentic Electronic Design Automation: A Handoff Perspective](https://mlsanity.com/paper/2606.19795/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated capabilities for producing code in Hardware Description Languages (HDLs). However, most of the focus remains on their abilities to write functional code, not test code. The hardware design process consists of both design and test, and so eschewing validation and verification leaves considerable potential benefit unexplored, given that a design and test framework may allow for progress towards full automation of the digital design pipeline. In this work, we perform one of the first studies exploring how a LLM can both design and test hardware modules from provided specifications. Using a suite of 8 representative benchmarks, we examined the capabilities and limitations of the state-of-the-art conversational LLMs when producing Verilog for functional and verification purposes. We taped out the benchmarks on a Skywater 130nm shuttle and received the functional chip.