mlsanityData to 7 Oct 2026

Benchmarking Large Language Models for Automated Verilog RTL Code Generation

arXiv 2212.11140 ↗First posted 13 Dec 2022cs.PL cs.LG cs.SEcited by 112

Not on the map. Why?

Abstract (arXiv)

Automating hardware design could obviate a significant amount of human error from the engineering process and lead to fewer errors. Verilog is a popular hardware description language to model and design digital systems, thus generating Verilog code is a critical first step. Emerging large language models (LLMs) are able to write high-quality code in other programming languages. In this paper, we characterize the ability of LLMs to generate useful Verilog. For this, we fine-tune pre-trained LLMs on Verilog datasets collected from GitHub and Verilog textbooks. We construct an evaluation framework comprising test-benches for functional analysis and a flow to test the syntax of Verilog code generated in response to problems of varying difficulty. Our findings show that across our problem scenarios, the fine-tuning results in LLMs more capable of producing syntactically correct code (25.9% overall). Further, when analyzing functional correctness, a fine-tuned open-source CodeGen LLM can outperform the state-of-the-art commercial Codex LLM (6.5% overall). Training/evaluation scripts and LLM checkpoints are available: https://github.com/shailja-thakur/VGen.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Builds on 9 of its 15 references that are in the map. First posted on arXiv 13 Dec 2022.

Builds on

9 of 15 references.

References · 9

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  8. 2020DAVE: Deriving Automatically Verilog from Englishcited by 46
  9. 2017PrivMin: Differentially Private MinHash for Jaccard Similarity Computationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 41; 2025: 36; 2026: 18.

112 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Models for Code Generationcited by 646
  2. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 246
  3. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 197
  4. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 168
  5. 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 143
  6. 2023Chip-Chat: Challenges and Opportunities in Conversational Hardware Designcited by 136
  7. 2024BetterV: Controlled Verilog Generation with Discriminative Guidancecited by 99
  8. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  9. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  10. 2023GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Modelscited by 59
  11. 2024OriGen:Enhancing RTL Code Generation with Code-to-Code Augmentation and Self-Reflectioncited by 56
  12. 2024MG-Verilog: Multi-grained Dataset Towards Enhanced LLM-assisted Verilog Generationcited by 45
38 more
  1. 2024Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generationcited by 45
  2. 2024ADO-LLM: Analog Design Bayesian Optimization with In-Context Learning of Large Language Modelscited by 44
  3. 2023LLM4EDA: Emerging Progress in Large Language Models for Electronic Design Automationcited by 40
  4. 2024CodeV: Empowering LLMs with HDL Generation through Multi-Level Summarizationcited by 38
  5. 2024AutoVCoder: A Systematic Framework for Automated Verilog Code Generation using LLMscited by 36
  6. 2023(Security) Assertions by Large Language Modelscited by 31
  7. 2024Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QAcited by 28
  8. 2024DeepGate3: Towards Scalable Circuit Representation Learningcited by 26
  9. 2024RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projectscited by 25
  10. 2024Rome was Not Built in a Single Step: Hierarchical Prompting for LLM-based Chip Designcited by 24
  11. 2025DeepRTL: Bridging Verilog Understanding and Generation with a Unified Representation Modelcited by 23
  12. 2023LLM for SoC Security: A Paradigm Shiftcited by 22
  13. 2023LLM4DV: Using Large Language Models for Hardware Test Stimuli Generationcited by 21
  14. 2025DeepCircuitX: A Comprehensive Repository-Level Dataset for RTL Code Understanding, Generation, and PPA Analysiscited by 21
  15. 2024CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generationcited by 19
  16. 2024PyraNet: A Multi-Layered Hierarchical Dataset for Verilogcited by 19
  17. 2024LLM4SecHW: Leveraging Domain Specific Large Language Model for Hardware Debuggingcited by 18
  18. 2024Automatically Improving LLM-based Verilog Generation using EDA Tool Feedbackcited by 18
  19. 2025HDLCoRe: A Training-Free Framework for Mitigating Hallucinations in LLM-Generated HDLcited by 17
  20. 2024AMSnet-KG: A Netlist Dataset for LLM-based AMS Circuit Auto-Design Using Knowledge Graph RAGcited by 16
  21. 2024A Multi-Expert Large Language Model Architecture for Verilog Code Generationcited by 15
  22. 2024Large Language Model for Verilog Generation with Code-Structure-Guided Reinforcement Learningcited by 15
  23. 2025VeriThoughts: Enabling Automated Verilog Code Generation using Reasoning and Formal Verificationcited by 15
  24. 2023Advanced Large Language Model (LLM)-Driven Verilog Development: Enhancing Power, Performance, and Area Optimization in Code Synthesiscited by 14
  25. 2024VHDL-Eval: A Framework for Evaluating Large Language Models in VHDL Code Generationcited by 14
  26. 2025RealBench: Benchmarking Verilog Generation Models with Real-World IP Designscited by 14
  27. 2024HiVeGen -- Hierarchical LLM-based Verilog Generation for Scalable Chip Designcited by 13
  28. 2025VeriDebug: A Unified LLM for Verilog Debugging via Contrastive Embedding and Guided Correctioncited by 12
  29. 2024All Artificial, Less Intelligence: GenAI through the Lens of Formal Verificationcited by 11
  30. 2024Natural language is not enough: Benchmarking multi-modal generative AI for Verilog generationcited by 11
  31. 2025VeriCoder: Enhancing LLM-Based RTL Code Generation through Functional Correctness Validationcited by 11
  32. 2023A Deep Learning Framework for Verilog Autocompletion Towards Design and Verification Automationcited by 10
  33. 2024Hardware Phi-1.5B: A Large Language Model Encodes Hardware Domain Specific Knowledgecited by 10
  34. 2023Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Codecited by 9
  35. 2024ITERTL: An Iterative Framework for Fine-tuning LLMs for RTL Code Generationcited by 9
  36. 2025MCP4EDA: LLM-Powered Model Context Protocol RTL-to-GDSII Automation with Backend Aware Synthesis Optimizationcited by 9
  37. 2024Evaluating LLMs for Hardware Design and Testcited by 8
  38. 2024Location is Key: Leveraging Large Language Model for Functional Bug Localization in Verilogcited by 8

50 most cited of 112 All 112 (JSON)