# Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification

arXiv: https://arxiv.org/abs/2506.14074 · HTML: https://mlsanity.com/paper/2506.14074/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 10 of its 33 references that are in the map.
- First posted on arXiv 17 Jun 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 10 of 10)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Understanding How Dimension Reduction Tools Work: An Empirical Approach to Deciphering t-SNE, UMAP, TriMAP, and PaCMAP for Data Visualization](https://mlsanity.com/paper/2012.04456/) (2020)
- [RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model](https://mlsanity.com/paper/2308.05345/) (2023)
- [VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Tool](https://mlsanity.com/paper/2408.08927/) (2024)
- [Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation](https://mlsanity.com/paper/2408.11053/) (2024)
- [CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair](https://mlsanity.com/paper/2409.12993/) (2024)
- [OpenLLM-RTL: Open Dataset and Benchmark for LLM-Aided Design RTL Generation](https://mlsanity.com/paper/2503.15112/) (2025)
- [LLM4DV: Using Large Language Models for Hardware Test Stimuli Generation](https://mlsanity.com/paper/2310.04535/) (2023)
- [Large Language Model for Verilog Generation with Code-Structure-Guided Reinforcement Learning](https://mlsanity.com/paper/2407.18271/) (2024)
- [Copilot Evaluation Harness: Evaluating LLM-Guided Software Programming](https://mlsanity.com/paper/2402.14261/) (2024)

## Built on by (most cited first, 30 of 44)

- [ACE-RTL: When Agentic Context Evolution Meets RTL-Specialized LLMs](https://mlsanity.com/paper/2602.10218/) (2026)
- [Revolution or Hype? Seeking the Limits of Large Models in Hardware Design](https://mlsanity.com/paper/2509.04905/) (2025)
- [Large Language Model for Verilog Code Generation: Literature Review and the Road Ahead](https://mlsanity.com/paper/2512.00020/) (2025)
- [Agentic Hardware Design as Repository-Level Code Evolution](https://mlsanity.com/paper/2606.28279/) (2026)
- [LLM4Cov: Execution-Aware Agentic Learning for High-coverage Testbench Generation](https://mlsanity.com/paper/2602.16953/) (2026)
- [TB or Not TB: Coverage-Driven Direct Preference Optimization for Verilog Stimulus Generation](https://mlsanity.com/paper/2511.15767/) (2025)
- [GRPO with State Mutations: Improving LLM-Based Hardware Test Plan Generation](https://mlsanity.com/paper/2601.07593/) (2026)
- [SiliconMind-V1: Multi-Agent Distillation and Debug-Reasoning Workflows for Verilog Code Generation](https://mlsanity.com/paper/2603.08719/) (2026)
- [QuArch: A Benchmark for Evaluating LLM Reasoning in Computer Architecture](https://mlsanity.com/paper/2510.22087/) (2025)
- [Trace2Skill: Verifier-Guided Skill Evolution for Long-Context EDA Agents](https://mlsanity.com/paper/2605.21810/) (2026)
- [Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents](https://mlsanity.com/paper/2510.17491/) (2025)
- [Bench4HLS: End-to-End Evaluation of LLMs in High-Level Synthesis Code Generation](https://mlsanity.com/paper/2601.19941/) (2026)
- [UCAgent: An End-to-End Agent for Block-Level Functional Verification](https://mlsanity.com/paper/2603.25768/) (2026)
- [SafeTune: Mitigating Data Poisoning in LLM Fine-Tuning for RTL Code Generation](https://mlsanity.com/paper/2604.27238/) (2026)
- [CRADLE: Conversational RTL Design Space Exploration with LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2508.08709/) (2025)
- [DeepV: A Model-Agnostic Retrieval-Augmented Framework for Verilog Code Generation with a High-Quality Knowledge Base](https://mlsanity.com/paper/2510.05327/) (2025)
- [Multimodal Chip Physical Design Engineer Assistant](https://mlsanity.com/paper/2510.15872/) (2025)
- [QiMeng-ChipV-RTL: Exploiting Information Locality for IP-level Verilog Generation](https://mlsanity.com/paper/2602.00704/) (2026)
- [Saarthi for AGI: Towards Domain-Specific General Intelligence for Formal Verification](https://mlsanity.com/paper/2603.03175/) (2026)
- [Exploring the Agentic Frontier of Verilog Code Generation](https://mlsanity.com/paper/2603.19347/) (2026)
- [ChipMATE: Multi-Agent Training via Reinforcement Learning for Enhanced RTL Generation](https://mlsanity.com/paper/2605.12857/) (2026)
- [NoTB: Oracle-Free Triage of LLM-Generated RTL via Cross-Model Formal Consensus](https://mlsanity.com/paper/2608.21962/) (2026)
- [LLMs in Digital EDA: A perspective on shifting roles from Generation to Orchestration](https://mlsanity.com/paper/2608.27184/) (2026)
- [Automatic Harness Evolution for Hardware Design Verification: Can LLMs Consolidate Gains Across Discovered Harnesses?](https://mlsanity.com/paper/2609.28908/) (2026)
- [Agentic AI-based Coverage Closure for Formal Verification](https://mlsanity.com/paper/2603.03147/) (2026)
- [ChipCraftBrain: Validation-First RTL Generation via Multi-Agent Orchestration](https://mlsanity.com/paper/2604.19856/) (2026)
- [RuC: HDL-Agnostic Rule Completion Benchmark Generation](https://mlsanity.com/paper/2604.27780/) (2026)
- [CktFormalizer: Autoformalization of Natural Language into Circuit Representations](https://mlsanity.com/paper/2605.07782/) (2026)
- [CASS-RTL: Correctness-Aware Subspace Steering for RTL Generation with LLMs](https://mlsanity.com/paper/2606.05680/) (2026)
- [HierSVA: A Data Synthesis Pipeline, Dataset, and Benchmark for LLM-Driven Hierarchical Hardware Formal Verification](https://mlsanity.com/paper/2606.13706/) (2026)

## Abstract (arXiv)

We present the Comprehensive Verilog Design Problems (CVDP) benchmark, a new dataset and infrastructure to advance LLM and agent research in hardware design and verification. CVDP includes 783 problems across 13 task categories, covering RTL generation, verification, debugging, specification alignment, and technical Q&A authored by experienced hardware engineers. Problems are offered in both non-agentic and agentic formats. The benchmark introduces more realistic and challenging contexts than prior work, with state-of-the-art models achieving no more than 34% pass@1 on code generation. Agentic tasksunicodex2013especially those involving RTL reuse and verificationunicodex2013are particularly difficult. Evaluation uses open-source tools and model scoring infrastructure, with comprehension tasks assessed via BLEU and LLM-based judging. CVDP reveals substantial gaps in current model capabilities, underscoring the need for continued research toward robust, real-world hardware design automation.