# Automatically Improving LLM-based Verilog Generation using EDA Tool Feedback

arXiv: https://arxiv.org/abs/2411.11856 · HTML: https://mlsanity.com/paper/2411.11856/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 45 references that are in the map.
- First posted on arXiv 1 Nov 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [ChipNeMo: Domain-Adapted LLMs for Chip Design](https://mlsanity.com/paper/2311.00176/) (2023)
- [Chip-Chat: Challenges and Opportunities in Conversational Hardware Design](https://mlsanity.com/paper/2305.13243/) (2023)
- [ChatEDA: A Large Language Model Powered Autonomous Agent for EDA](https://mlsanity.com/paper/2308.10204/) (2023)
- [Benchmarking Large Language Models for Automated Verilog RTL Code Generation](https://mlsanity.com/paper/2212.11140/) (2022)
- [ChipGPT: How far are we from natural language hardware design](https://mlsanity.com/paper/2305.14019/) (2023)
- [BetterV: Controlled Verilog Generation with Discriminative Guidance](https://mlsanity.com/paper/2402.03375/) (2024)
- [AutoChip: Automating HDL Generation Using LLM Feedback](https://mlsanity.com/paper/2311.04887/) (2023)
- [RTLCoder: Outperforming GPT-3.5 in Design RTL Generation with Our Open-Source Dataset and Lightweight Solution](https://mlsanity.com/paper/2312.08617/) (2023)
- [GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Models](https://mlsanity.com/paper/2309.10730/) (2023)
- [DAVE: Deriving Automatically Verilog from English](https://mlsanity.com/paper/2009.01026/) (2020)
- [Make Every Move Count: LLM-based High-Quality RTL Code Generation Using MCTS](https://mlsanity.com/paper/2402.03289/) (2024)
- [Towards LLM-Powered Verilog RTL Assistant: Self-Verification and Self-Correction](https://mlsanity.com/paper/2406.00115/) (2024)
- [(Security) Assertions by Large Language Models](https://mlsanity.com/paper/2306.14027/) (2023)
- [Do Large Language Models Rank Fairly? An Empirical Study on the Fairness of LLMs as Rankers](https://mlsanity.com/paper/2404.03192/) (2024)
- [C2HLSC: Can LLMs Bridge the Software-to-Hardware Design Gap?](https://mlsanity.com/paper/2406.09233/) (2024)
- [Evaluating LLMs for Hardware Design and Test](https://mlsanity.com/paper/2405.02326/) (2024)

## Built on by (most cited first, 19 of 19)

- [A Survey of Circuit Foundation Model: Foundation AI Models for VLSI Circuit Design and EDA](https://mlsanity.com/paper/2504.03711/) (2025)
- [Large Language Model for Verilog Code Generation: Literature Review and the Road Ahead](https://mlsanity.com/paper/2512.00020/) (2025)
- [Towards LLM-based Root Cause Analysis of Hardware Design Failures](https://mlsanity.com/paper/2507.06512/) (2025)
- [Large Language Models (LLMs) for Electronic Design Automation (EDA)](https://mlsanity.com/paper/2508.20030/) (2025)
- [SANGAM: SystemVerilog Assertion Generation via Monte Carlo Tree Self-Refine](https://mlsanity.com/paper/2506.13983/) (2025)
- [QiMeng-CRUX: Narrowing the Gap Between Natural Language and Verilog via Core Refined Understanding eXpression for Circuit Design](https://mlsanity.com/paper/2511.20099/) (2025)
- [GUIDE: GenAI Units In Digital Design Education](https://mlsanity.com/paper/2603.17296/) (2026)
- [LLMs for Secure Hardware Design and Related Problems: Opportunities and Challenges](https://mlsanity.com/paper/2605.10807/) (2026)
- [SABLE: An NDA-Safe Closed-Loop LLM Framework for Analog Circuit Optimization in Industrial EDA Flows](https://mlsanity.com/paper/2607.03701/) (2026)
- [LLMs in Digital EDA: A perspective on shifting roles from Generation to Orchestration](https://mlsanity.com/paper/2608.27184/) (2026)
- [Lorecast: Layout-Aware Performance and Power Forecasting from Natural Language](https://mlsanity.com/paper/2503.11662/) (2025)
- [Mitigating hallucinations and omissions in LLMs for invertible problems: An application to hardware logic design automation](https://mlsanity.com/paper/2512.03053/) (2025)
- [VHDLSuite: Unified Pipeline for LLM VHDL Generation with Data Synthesis and Evaluation](https://mlsanity.com/paper/2606.13735/) (2026)
- [Agentic Electronic Design Automation: A Handoff Perspective](https://mlsanity.com/paper/2606.19795/) (2026)
- [PCBWorld: A Benchmark Environment for Engine-Grounded PCB Design Automation](https://mlsanity.com/paper/2607.05915/) (2026)
- [EDATracer: An Agentic Framework for Large-Scale EDA Artifact Analysis](https://mlsanity.com/paper/2608.04032/) (2026)
- [Hardware Design and Security in the Era of Chiplets and LLMs](https://mlsanity.com/paper/2608.05063/) (2026)
- [SeqFeed: Improving Agentic RTL Code Generation with Sequential Behavior Feedback](https://mlsanity.com/paper/2608.16934/) (2026)
- [SparseCraft: Agentic Hardware-Software Co-Optimization for Sparse Computing](https://mlsanity.com/paper/2610.05037/) (2026)

## Abstract (arXiv)

Traditionally, digital hardware designs are written in the Verilog hardware description language (HDL) and debugged manually by engineers. This can be time-consuming and error-prone for complex designs. Large Language Models (LLMs) are emerging as a potential tool to help generate fully functioning HDL code, but most works have focused on generation in the single-shot capacity: i.e., run and evaluate, a process that does not leverage debugging and, as such, does not adequately reflect a realistic development process. In this work, we evaluate the ability of LLMs to leverage feedback from electronic design automation (EDA) tools to fix mistakes in their own generated Verilog. To accomplish this, we present an open-source, highly customizable framework, AutoChip, which combines conversational LLMs with the output from Verilog compilers and simulations to iteratively generate and repair Verilog. To determine the success of these LLMs we leverage the VerilogEval benchmark set. We evaluate four state-of-the-art conversational LLMs, focusing on readily accessible commercial models. EDA tool feedback proved to be consistently more effective than zero-shot prompting only with GPT-4o, the most computationally complex model we evaluated. In the best case, we observed a 5.8% increase in the number of successful designs with a 34.2% decrease in cost over the best zero-shot results. Mixing smaller models with this larger model at the end of the feedback iterations resulted in equally as much success as with GPT-4o using feedback, but incurred 41.9% lower cost (corresponding to an overall decrease in cost over zero-shot by 89.6%).