mlsanityData to 7 Oct 2026

VerilogEval: Evaluating Large Language Models for Verilog Code Generation

arXiv 2309.07544 ↗First posted 14 Sep 2023cs.LG cs.SEcited by 246

Not on the map. Why?

Cited by 50 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The increasing popularity of large language models (LLMs) has paved the way for their application in diverse domains. This paper proposes a benchmarking framework tailored specifically for evaluating LLM performance in the context of Verilog code generation for hardware design and verification. We present a comprehensive evaluation dataset consisting of 156 problems from the Verilog instructional website HDLBits. The evaluation set consists of a diverse set of Verilog code generation tasks, ranging from simple combinational circuits to complex finite state machines. The Verilog code completions can be automatically tested for functional correctness by comparing the transient simulation outputs of the generated design with a golden solution. We also demonstrate that the Verilog code generation capability of pretrained language models could be improved with supervised fine-tuning by bootstrapping with LLM generated synthetic problem-code pairs.

Key facts. Cited by 246 papers in the map (data through 7 Oct 2026). Builds on 28 of its 34 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

28 of 34 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
16 more
  1. 2023BloombergGPT: A Large Language Model for Financecited by 923
  2. 2021Measuring Coding Challenge Competence With APPScited by 879
  3. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  4. 2022Galactica: A Large Language Model for Sciencecited by 736
  5. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  6. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  7. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  8. 2023Textbooks Are All You Needcited by 561
  9. 2019SPoC: Search-based Pseudocode to Codecited by 176
  10. 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 143
  11. 2023Chip-Chat: Challenges and Opportunities in Conversational Hardware Designcited by 136
  12. 2022Benchmarking Large Language Models for Automated Verilog RTL Code Generationcited by 112
  13. 2023ChipGPT: How far are we from natural language hardware designcited by 101
  14. 2020BioMegatron: Larger Biomedical Domain Language Modelcited by 76
  15. 2023On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilotcited by 64
  16. 2023CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Modelscited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 47; 2025: 97; 2026: 96.

246 papers.

Sort:

By area:

  1. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 168
  2. 2024BetterV: Controlled Verilog Generation with Discriminative Guidancecited by 99
  3. 2024AnalogCoder: Analog Circuit Design via Training-Free Code Generationcited by 82
  4. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 80
  5. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  6. 2024OriGen:Enhancing RTL Code Generation with Code-to-Code Augmentation and Self-Reflectioncited by 56
  7. 2024MG-Verilog: Multi-grained Dataset Towards Enhanced LLM-assisted Verilog Generationcited by 45
  8. 2024Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generationcited by 45
  9. 2024ADO-LLM: Analog Design Bayesian Optimization with In-Context Learning of Large Language Modelscited by 44
  10. 2024Make Every Move Count: LLM-based High-Quality RTL Code Generation Using MCTScited by 42
  11. 2024CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repaircited by 41
  12. 2023LLM4EDA: Emerging Progress in Large Language Models for Electronic Design Automationcited by 40
38 more
  1. 2024CodeV: Empowering LLMs with HDL Generation through Multi-Level Summarizationcited by 38
  2. 2025A Survey of Research in Large Language Models for Electronic Design Automationcited by 38
  3. 2024Can Large Language Models Write Parallel Code?cited by 36
  4. 2024AutoVCoder: A Systematic Framework for Automated Verilog Code Generation using LLMscited by 36
  5. 2025AnalogGenie: A Generative Engine for Automatic Discovery of Analog Circuit Topologiescited by 35
  6. 2024RTLRewriter: Methodologies for Large Models aided RTL Code Optimizationcited by 30
  7. 2024Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QAcited by 28
  8. 2024HDLdebugger: Streamlining HDL debugging with Large Language Modelscited by 25
  9. 2024RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projectscited by 25
  10. 2024Rome was Not Built in a Single Step: Hierarchical Prompting for LLM-based Chip Designcited by 24
  11. 2025DeepRTL: Bridging Verilog Understanding and Generation with a Unified Representation Modelcited by 23
  12. 2023LLM for SoC Security: A Paradigm Shiftcited by 22
  13. 2025ScaleRTL: Scaling LLMs with Reasoning Data and Test-Time Compute for Accurate RTL Code Generationcited by 22
  14. 2025DeepCircuitX: A Comprehensive Repository-Level Dataset for RTL Code Understanding, Generation, and PPA Analysiscited by 21
  15. 2024CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generationcited by 19
  16. 2025VFlow: Discovering Optimal Agentic Workflows for Verilog Generationcited by 19
  17. 2024LLMs and the Future of Chip Design: Unveiling Security Risks and Building Trustcited by 18
  18. 2024Automatically Improving LLM-based Verilog Generation using EDA Tool Feedbackcited by 18
  19. 2024AssertionBench: A Benchmark to Evaluate Large-Language Models for Assertion Generationcited by 16
  20. 2025SymRTLO: Enhancing RTL Code Optimization with LLMs and Neuron-Inspired Symbolic Reasoningcited by 16
  21. 2025VeriReason: Reinforcement Learning with Testbench Feedback for Reasoning-Enhanced Verilog Generationcited by 16
  22. 2025RTL++: Graph-enhanced LLM for RTL Code Generationcited by 16
  23. 2024A Multi-Expert Large Language Model Architecture for Verilog Code Generationcited by 15
  24. 2024Large Language Model for Verilog Generation with Code-Structure-Guided Reinforcement Learningcited by 15
  25. 2024AIvril: AI-Driven RTL Generation With Verification In-The-Loopcited by 15
  26. 2025VeriThoughts: Enabling Automated Verilog Code Generation using Reasoning and Formal Verificationcited by 15
  27. 2023Advanced Large Language Model (LLM)-Driven Verilog Development: Enhancing Power, Performance, and Area Optimization in Code Synthesiscited by 14
  28. 2024VHDL-Eval: A Framework for Evaluating Large Language Models in VHDL Code Generationcited by 14
  29. 2025Insights from Verification: Training a Verilog Generation LLM with Reinforcement Learning with Testbench Feedbackcited by 14
  30. 2025AMSbench: A Comprehensive Benchmark for Evaluating MLLM Capabilities in AMS Circuitscited by 14
  31. 2025RealBench: Benchmarking Verilog Generation Models with Real-World IP Designscited by 14
  32. 2024HiVeGen -- Hierarchical LLM-based Verilog Generation for Scalable Chip Designcited by 13
  33. 2024EDA Corpus: A Large Language Model Dataset for Enhanced Interaction with OpenROADcited by 12
  34. 2025VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contaminationcited by 12
  35. 2025ReasoningV: Efficient Verilog Code Generation with Adaptive Hybrid Reasoning Modelcited by 12
  36. 2025VeriDebug: A Unified LLM for Verilog Debugging via Contrastive Embedding and Guided Correctioncited by 12
  37. 2026ACE-RTL: When Agentic Context Evolution Meets RTL-Specialized LLMscited by 12
  38. 2024LLM-Enhanced Bayesian Optimization for Efficient Analog Layout Constraint Generationcited by 11

50 most cited of 246 All 246 (JSON)