mlsanityData to 7 Oct 2026

Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation

arXiv 2408.11053 ↗First posted 20 Aug 2024cs.AR cs.AIcited by 45

Not on the map. Why?

Abstract (arXiv)

The application of large-language models (LLMs) to digital hardware code generation is an emerging field, with most LLMs primarily trained on natural language and software code. Hardware code like Verilog constitutes a small portion of training data, and few hardware benchmarks exist. The open-source VerilogEval benchmark, released in November 2023, provided a consistent evaluation framework for LLMs on code completion tasks. Since then, both commercial and open models have seen significant development. In this work, we evaluate new commercial and open models since VerilogEval's original release-including GPT-4o, GPT-4 Turbo, Llama3.1 (8B/70B/405B), Llama3 70B, Mistral Large, DeepSeek Coder (33B and 6.7B), CodeGemma 7B, and RTL-Coder-against an improved VerilogEval benchmark suite. We find measurable improvements in state-of-the-art models: GPT-4o achieves a 63% pass rate on specification-to-RTL tasks. The recently released and open Llama3.1 405B achieves a 58% pass rate, almost matching GPT-4o, while the smaller domain-specific RTL-Coder 6.7B models achieve an impressive 34% pass rate. Additionally, we enhance VerilogEval's infrastructure by automatically classifying failures, introducing in-context learning support, and extending the tasks to specification-to-RTL translation. We find that prompt engineering remains crucial for achieving good pass rates and varies widely with model and task. A benchmark infrastructure that allows for prompt engineering and failure analysis is essential for continued model development and deployment.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 18 of its 35 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

18 of 35 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  4. 2022Transformers learn in-context by gradient descentcited by 714
  5. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  6. 2023In-Context Learning Creates Task Vectorscited by 302
  7. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 246
  8. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  9. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 197
  10. 2023Chip-Chat: Challenges and Opportunities in Conversational Hardware Designcited by 136
  11. 2022Benchmarking Large Language Models for Automated Verilog RTL Code Generationcited by 112
  12. 2023ChipGPT: How far are we from natural language hardware designcited by 101
6 more
  1. 2024VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Toolcited by 85
  2. 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 49
  3. 2024Towards LLM-Powered Verilog RTL Assistant: Self-Verification and Self-Correctioncited by 39
  4. 2024RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projectscited by 25
  5. 2023LLM4DV: Using Large Language Models for Hardware Test Stimuli Generationcited by 21
  6. 2021AutoSVA: Democratizing Formal Verification of RTL Module Interactionscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 22; 2026: 22.

45 papers.

Sort:

By area:

  1. 2025Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verificationcited by 41
  2. 2025QiMeng-CodeV-R1: Reasoning-Enhanced Verilog Generationcited by 34
  3. 2024MAGE: A Multi-Agent Engine for Automated RTL Code Generationcited by 31
  4. 2025VFlow: Discovering Optimal Agentic Workflows for Verilog Generationcited by 19
  5. 2025HDLCoRe: A Training-Free Framework for Mitigating Hallucinations in LLM-Generated HDLcited by 17
  6. 2025VeriThoughts: Enabling Automated Verilog Code Generation using Reasoning and Formal Verificationcited by 15
  7. 2025VeriMoA: A Mixture-of-Agents Framework for Spec-to-HDL Generationcited by 15
  8. 2025Insights from Verification: Training a Verilog Generation LLM with Reinforcement Learning with Testbench Feedbackcited by 14
  9. 2025ReasoningV: Efficient Verilog Code Generation with Adaptive Hybrid Reasoning Modelcited by 12
  10. 2026POET: Power-Oriented Evolutionary Tuning for LLM-Based RTL PPA Optimizationcited by 11
  11. 2025Revolution or Hype? Seeking the Limits of Large Models in Hardware Designcited by 9
  12. 2025TuRTLe: A Unified Evaluation of LLMs for RTL Generationcited by 8
33 more
  1. 2025Large Language Model for Verilog Code Generation: Literature Review and the Road Aheadcited by 7
  2. 2026ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Designcited by 7
  3. 2025PRO-V-R1: Reasoning Enhanced Programming Agent for RTL Verificationcited by 6
  4. 2025LLM-VeriPPA: Power, Performance, and Area Optimization aware Verilog Code Generation with Large Language Modelscited by 5
  5. 2026HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generationcited by 5
  6. 2026Forgetting in Language Models: Capacity, Optimization, and Self-Generated Replaycited by 5
  7. 2025Towards LLM-based Root Cause Analysis of Hardware Design Failurescited by 4
  8. 2025REvolution: An Evolutionary Framework for RTL Generation driven by Large Language Modelscited by 4
  9. 2026SiliconMind-V1: Multi-Agent Distillation and Debug-Reasoning Workflows for Verilog Code Generationcited by 4
  10. 2025NotSoTiny: A Large, Living Benchmark for RTL Code Generationcited by 3
  11. 2026Veri-Sure: A Contract-Aware Multi-Agent Framework with Temporal Tracing and Formal Verification for Correct RTL Code Generationcited by 3
  12. 2026COEVO: Co-Evolutionary Framework for Joint Functional Correctness and PPA Optimization in LLM-Based RTL Generationcited by 3
  13. 2025hdl2v: A Code Translation Dataset for Enhanced LLM Verilog Generationcited by 2
  14. 2025EARL: Entropy-Aware RL Alignment of LLMs for Reliable RTL Code Generationcited by 2
  15. 2025Assessing Large Language Models in Generating RTL Design Specificationscited by 2
  16. 2026Bench4HLS: End-to-End Evaluation of LLMs in High-Level Synthesis Code Generationcited by 2
  17. 2026LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Reviewcited by 2
  18. 2025DeepV: A Model-Agnostic Retrieval-Augmented Framework for Verilog Code Generation with a High-Quality Knowledge Basecited by 1
  19. 2025QiMeng-CRUX: Narrowing the Gap Between Natural Language and Verilog via Core Refined Understanding eXpression for Circuit Designcited by 1
  20. 2026LAUDE: LLM-Assisted Unit Test Generation and Debugging of Hardware DEsignscited by 1
  21. 2026VeriInteresting: An Empirical Study of Model Prompt Interactions in Verilog Code Generationcited by 1
  22. 2026Configuration Over Selection: Hyperparameter Sensitivity Exceeds Model Differences in Open-Source LLMs for RTL Generationcited by 1
  23. 2026ChipMATE: Multi-Agent Training via Reinforcement Learning for Enhanced RTL Generationcited by 1
  24. 2026Structured Testbench Generation for LLM-Driven HDL Design and Verification-Oriented Data Curationcited by 1
  25. 2025Lorecast: Layout-Aware Performance and Power Forecasting from Natural Languagecited by 0
  26. 2026CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Languagecited by 0
  27. 2026Exploring LLM-based Verilog Code Generation with Data-Efficient Fine-Tuning and Testbench Automationcited by 0
  28. 2026AutoPPA: Automated Circuit PPA Optimization via Contrastive Code-based Rule Library Learningcited by 0
  29. 2026From Mirage to Grounding: Towards Reliable Multimodal Circuit-to-Verilog Code Generationcited by 0
  30. 2026VeriTrace: Human-Like Temporal Exploration Completes Agentic Action Spacecited by 0
  31. 2026NoTB: Oracle-Free Triage of LLM-Generated RTL via Cross-Model Formal Consensuscited by 0
  32. 2026TicTacBench: Benchmarking Timing Closure Capabilities of Coding Agentscited by 0
  33. 2026Automatic Harness Evolution for Hardware Design Verification: Can LLMs Consolidate Gains Across Discovered Harnesses?cited by 0