# OpenLLM-RTL: Open Dataset and Benchmark for LLM-Aided Design RTL Generation

arXiv: https://arxiv.org/abs/2503.15112 · HTML: https://mlsanity.com/paper/2503.15112/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2025.

## Built on by (most cited first, 30 of 31)

- [Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification](https://mlsanity.com/paper/2506.14074/) (2025)
- [VeriMoA: A Mixture-of-Agents Framework for Spec-to-HDL Generation](https://mlsanity.com/paper/2510.27617/) (2025)
- [Insights from Verification: Training a Verilog Generation LLM with Reinforcement Learning with Testbench Feedback](https://mlsanity.com/paper/2504.15804/) (2025)
- [RealBench: Benchmarking Verilog Generation Models with Real-World IP Designs](https://mlsanity.com/paper/2507.16200/) (2025)
- [VeriCoder: Enhancing LLM-Based RTL Code Generation through Functional Correctness Validation](https://mlsanity.com/paper/2504.15659/) (2025)
- [ReasoningV: Efficient Verilog Code Generation with Adaptive Hybrid Reasoning Model](https://mlsanity.com/paper/2504.14560/) (2025)
- [VerilogDB: The Largest, Highest-Quality Dataset with a Preprocessing Framework for LLM-based RTL Generation](https://mlsanity.com/paper/2507.13369/) (2025)
- [Large Language Model for Verilog Code Generation: Literature Review and the Road Ahead](https://mlsanity.com/paper/2512.00020/) (2025)
- [ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design](https://mlsanity.com/paper/2601.21448/) (2026)
- [EvolVE: Evolutionary Search for LLM-based Verilog Generation and Optimization](https://mlsanity.com/paper/2601.18067/) (2026)
- [ReChisel: Effective Automatic Chisel Code Generation by LLM with Reflection](https://mlsanity.com/paper/2505.19734/) (2025)
- [MMCircuitEval: A Comprehensive Multimodal Circuit-Focused Benchmark for Evaluating LLMs](https://mlsanity.com/paper/2507.19525/) (2025)
- [Agentic Hardware Design as Repository-Level Code Evolution](https://mlsanity.com/paper/2606.28279/) (2026)
- [QiMeng-SALV: Signal-Aware Learning for Verilog Code Generation](https://mlsanity.com/paper/2510.19296/) (2025)
- [REvolution: An Evolutionary Framework for RTL Generation driven by Large Language Models](https://mlsanity.com/paper/2510.21407/) (2025)
- [SiliconMind-V1: Multi-Agent Distillation and Debug-Reasoning Workflows for Verilog Code Generation](https://mlsanity.com/paper/2603.08719/) (2026)
- [Free and Fair Hardware: A Pathway to Copyright Infringement-Free Verilog Generation using LLMs](https://mlsanity.com/paper/2505.06096/) (2025)
- [EvoVerilog: Large Langugage Model Assisted Evolution of Verilog Code](https://mlsanity.com/paper/2508.13156/) (2025)
- [SafeTune: Mitigating Data Poisoning in LLM Fine-Tuning for RTL Code Generation](https://mlsanity.com/paper/2604.27238/) (2026)
- [Alpha-RTL: Test-Time Training for RTL Hardware Optimization](https://mlsanity.com/paper/2606.05253/) (2026)
- [DeepV: A Model-Agnostic Retrieval-Augmented Framework for Verilog Code Generation with a High-Quality Knowledge Base](https://mlsanity.com/paper/2510.05327/) (2025)
- [Configuration Over Selection: Hyperparameter Sensitivity Exceeds Model Differences in Open-Source LLMs for RTL Generation](https://mlsanity.com/paper/2604.17102/) (2026)
- [ChipMATE: Multi-Agent Training via Reinforcement Learning for Enhanced RTL Generation](https://mlsanity.com/paper/2605.12857/) (2026)
- [AutoPPA: Automated Circuit PPA Optimization via Contrastive Code-based Rule Library Learning](https://mlsanity.com/paper/2604.18445/) (2026)
- [LEGO: An LLM Skill-Based Front-End Design Generation Platform](https://mlsanity.com/paper/2604.23355/) (2026)
- [From Mirage to Grounding: Towards Reliable Multimodal Circuit-to-Verilog Code Generation](https://mlsanity.com/paper/2604.27969/) (2026)
- [HierSVA: A Data Synthesis Pipeline, Dataset, and Benchmark for LLM-Driven Hierarchical Hardware Formal Verification](https://mlsanity.com/paper/2606.13706/) (2026)
- [Agentic Electronic Design Automation: A Handoff Perspective](https://mlsanity.com/paper/2606.19795/) (2026)
- [RTLCurator: Label-Efficient Data Curation for RTL Generation](https://mlsanity.com/paper/2607.29283/) (2026)
- [PerfReasoning: How Well Do LLMs Reason on Hardware Performance?](https://mlsanity.com/paper/2609.04476/) (2026)

## Abstract (arXiv)

The automated generation of design RTL based on large language model (LLM) and natural language instructions has demonstrated great potential in agile circuit design. However, the lack of datasets and benchmarks in the public domain prevents the development and fair evaluation of LLM solutions. This paper highlights our latest advances in open datasets and benchmarks from three perspectives: (1) RTLLM 2.0, an updated benchmark assessing LLM's capability in design RTL generation. The benchmark is augmented to 50 hand-crafted designs. Each design provides the design description, test cases, and a correct RTL code. (2) AssertEval, an open-source benchmark assessing the LLM's assertion generation capabilities for RTL verification. The benchmark includes 18 designs, each providing specification, signal definition, and correct RTL code. (3) RTLCoder-Data, an extended open-source dataset with 80K instruction-code data samples. Moreover, we propose a new verification-based method to verify the functionality correctness of training data samples. Based on this technique, we further release a dataset with 7K verified high-quality samples. These three studies are integrated into one framework, providing off-the-shelf support for the development and evaluation of LLMs for RTL code generation and verification. Finally, extensive experiments indicate that LLM performance can be boosted by enlarging the training dataset, improving data quality, and improving the training scheme.