mlsanityData to 14 Oct 2026

RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model

arXiv 2308.05345 ↗First posted 10 Aug 2023cs.LG cs.ARcited by 148

Not on the map. Why?

Cited by 30 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Inspired by the recent success of large language models (LLMs) like ChatGPT, researchers start to explore the adoption of LLMs for agile hardware design, such as generating design RTL based on natural-language instructions. However, in existing works, their target designs are all relatively simple and in a small scale, and proposed by the authors themselves, making a fair comparison among different LLM solutions challenging. In addition, many prior works only focus on the design correctness, without evaluating the design qualities of generated design RTL. In this work, we propose an open-source benchmark named RTLLM, for generating design RTL with natural language instructions. To systematically evaluate the auto-generated design RTL, we summarized three progressive goals, named syntax goal, functionality goal, and design quality goal. This benchmark can automatically provide a quantitative evaluation of any given LLM-based solution. Furthermore, we propose an easy-to-use yet surprisingly effective prompt engineering technique named self-planning, which proves to significantly boost the performance of GPT-3.5 in our proposed benchmark.

Key facts. Cited by 148 papers in the map (data through 14 Oct 2026). Builds on 8 of its 15 references that are in the map. First posted on arXiv 10 Aug 2023.

Builds on

8 of 15 references.

References · 8

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  4. 2023StarCoder: may the source be with you!cited by 799
  5. 2023Chip-Chat: Challenges and Opportunities in Conversational Hardware Designcited by 140
  6. 2022Benchmarking Large Language Models for Automated Verilog RTL Code Generationcited by 115
  7. 2023ChipGPT: How far are we from natural language hardware designcited by 105
  8. 2021Machine Learning for Electronic Design Automation: A Surveycited by 72

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 24; 2025: 63; 2026: 56.

148 papers.

Sort:

By area:

  1. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 252
  2. 2024BetterV: Controlled Verilog Generation with Discriminative Guidancecited by 100
  3. 2024AnalogCoder: Analog Circuit Design via Training-Free Code Generationcited by 84
  4. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  5. 2024MG-Verilog: Multi-grained Dataset Towards Enhanced LLM-assisted Verilog Generationcited by 46
  6. 2025Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verificationcited by 44
  7. 2024CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repaircited by 42
  8. 2023LLM4EDA: Emerging Progress in Large Language Models for Electronic Design Automationcited by 40
  9. 2024CodeV: Empowering LLMs with HDL Generation through Multi-Level Summarizationcited by 38
  10. 2024AutoVCoder: A Systematic Framework for Automated Verilog Code Generation using LLMscited by 36
  11. 2024Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QAcited by 28
  12. 2024RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projectscited by 26
38 more
  1. 2024AMSNet: Netlist Dataset for AMS Circuitscited by 25
  2. 2023LLM4DV: Using Large Language Models for Hardware Test Stimuli Generationcited by 23
  3. 2025DeepRTL: Bridging Verilog Understanding and Generation with a Unified Representation Modelcited by 23
  4. 2023LLM for SoC Security: A Paradigm Shiftcited by 22
  5. 2025ScaleRTL: Scaling LLMs with Reasoning Data and Test-Time Compute for Accurate RTL Code Generationcited by 22
  6. 2025DeepCircuitX: A Comprehensive Repository-Level Dataset for RTL Code Understanding, Generation, and PPA Analysiscited by 21
  7. 2024LLMs and the Future of Chip Design: Unveiling Security Risks and Building Trustcited by 20
  8. 2024CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generationcited by 19
  9. 2024PyraNet: A Multi-Layered Hierarchical Dataset for Verilogcited by 19
  10. 2025VFlow: Discovering Optimal Agentic Workflows for Verilog Generationcited by 19
  11. 2025RTL++: Graph-enhanced LLM for RTL Code Generationcited by 18
  12. 2025HDLCoRe: A Training-Free Framework for Mitigating Hallucinations in LLM-Generated HDLcited by 17
  13. 2025VeriReason: Reinforcement Learning with Testbench Feedback for Reasoning-Enhanced Verilog Generationcited by 16
  14. 2024VHDL-Eval: A Framework for Evaluating Large Language Models in VHDL Code Generationcited by 15
  15. 2024RTL-Breaker: Assessing the Security of LLMs against Backdoor Attacks on HDL Code Generationcited by 15
  16. 2025VeriThoughts: Enabling Automated Verilog Code Generation using Reasoning and Formal Verificationcited by 15
  17. 2025AMSbench: A Comprehensive Benchmark for Evaluating MLLM Capabilities in AMS Circuitscited by 15
  18. 2025VeriMoA: A Mixture-of-Agents Framework for Spec-to-HDL Generationcited by 15
  19. 2025Insights from Verification: Training a Verilog Generation LLM with Reinforcement Learning with Testbench Feedbackcited by 14
  20. 2025RealBench: Benchmarking Verilog Generation Models with Real-World IP Designscited by 14
  21. 2024LLM-Enhanced Bayesian Optimization for Efficient Analog Layout Constraint Generationcited by 13
  22. 2024HiVeGen -- Hierarchical LLM-based Verilog Generation for Scalable Chip Designcited by 13
  23. 2025VeriCoder: Enhancing LLM-Based RTL Code Generation through Functional Correctness Validationcited by 13
  24. 2025VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contaminationcited by 12
  25. 2025ReasoningV: Efficient Verilog Code Generation with Adaptive Hybrid Reasoning Modelcited by 12
  26. 2025SALAD: Systematic Assessment of Machine Unlearning on LLM-Aided Hardware Designcited by 12
  27. 2026ACE-RTL: When Agentic Context Evolution Meets RTL-Specialized LLMscited by 12
  28. 2024All Artificial, Less Intelligence: GenAI through the Lens of Formal Verificationcited by 11
  29. 2025VeriLeaky: Navigating IP Protection vs Utility in Fine-Tuning for LLM-Driven Verilog Codingcited by 11
  30. 2024Location is Key: Leveraging Large Language Model for Functional Bug Localization in Verilogcited by 10
  31. 2024SPICEPilot: Navigating SPICE Code Generation and Simulation with AI Guidancecited by 10
  32. 2025TuRTLe: A Unified Evaluation of LLMs for RTL Generationcited by 9
  33. 2025From Concept to Practice: an Automated LLM-aided UVM Machine for RTL Verificationcited by 9
  34. 2025Circuit Representation Learning with Masked Gate Modeling and Verilog-AIG Alignmentcited by 8
  35. 2025Large Language Model for Verilog Code Generation: Literature Review and the Road Aheadcited by 8
  36. 2026ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Designcited by 8
  37. 2024MetRex: A Benchmark for Verilog Code Metric Reasoning Using LLMscited by 7
  38. 2025Image2Net: Datasets, Benchmark and Hybrid Framework to Convert Analog Circuit Diagrams into Netlistscited by 7

50 most cited of 148 All 148 (JSON)