mlsanityData to 14 Oct 2026

Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification

arXiv 2506.14074 ↗First posted 17 Jun 2025cs.LG cs.ARcited by 44

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

We present the Comprehensive Verilog Design Problems (CVDP) benchmark, a new dataset and infrastructure to advance LLM and agent research in hardware design and verification. CVDP includes 783 problems across 13 task categories, covering RTL generation, verification, debugging, specification alignment, and technical Q&A authored by experienced hardware engineers. Problems are offered in both non-agentic and agentic formats. The benchmark introduces more realistic and challenging contexts than prior work, with state-of-the-art models achieving no more than 34% pass@1 on code generation. Agentic tasksunicodex2013especially those involving RTL reuse and verificationunicodex2013are particularly difficult. Evaluation uses open-source tools and model scoring infrastructure, with comprehension tasks assessed via BLEU and LLM-based judging. CVDP reveals substantial gaps in current model capabilities, underscoring the need for continued research toward robust, real-world hardware design automation.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 10 of its 33 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

10 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Tool15 Aug

Other references · 9

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  2. 2020Understanding How Dimension Reduction Tools Work: An Empirical Approach to Deciphering t-SNE, UMAP, TriMAP, and PaCMAP for Data Visualizationcited by 173
  3. 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 148
  4. 2024Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generationcited by 45
  5. 2024CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repaircited by 42
  6. 2025OpenLLM-RTL: Open Dataset and Benchmark for LLM-Aided Design RTL Generationcited by 31
  7. 2023LLM4DV: Using Large Language Models for Hardware Test Stimuli Generationcited by 23
  8. 2024Large Language Model for Verilog Generation with Code-Structure-Guided Reinforcement Learningcited by 15
  9. 2024Copilot Evaluation Harness: Evaluating LLM-Guided Software Programmingcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 36.

44 papers.

Sort:

By area:

  1. 2026ACE-RTL: When Agentic Context Evolution Meets RTL-Specialized LLMscited by 12
  2. 2025Revolution or Hype? Seeking the Limits of Large Models in Hardware Designcited by 9
  3. 2025Large Language Model for Verilog Code Generation: Literature Review and the Road Aheadcited by 8
  4. 2026Agentic Hardware Design as Repository-Level Code Evolutioncited by 6
  5. 2026LLM4Cov: Execution-Aware Agentic Learning for High-coverage Testbench Generationcited by 5
  6. 2025TB or Not TB: Coverage-Driven Direct Preference Optimization for Verilog Stimulus Generationcited by 4
  7. 2026GRPO with State Mutations: Improving LLM-Based Hardware Test Plan Generationcited by 4
  8. 2026SiliconMind-V1: Multi-Agent Distillation and Debug-Reasoning Workflows for Verilog Code Generationcited by 4
  9. 2025QuArch: A Benchmark for Evaluating LLM Reasoning in Computer Architecturecited by 3
  10. 2026Trace2Skill: Verifier-Guided Skill Evolution for Long-Context EDA Agentscited by 3
  11. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  12. 2026Bench4HLS: End-to-End Evaluation of LLMs in High-Level Synthesis Code Generationcited by 2
32 more
  1. 2026UCAgent: An End-to-End Agent for Block-Level Functional Verificationcited by 2
  2. 2026SafeTune: Mitigating Data Poisoning in LLM Fine-Tuning for RTL Code Generationcited by 2
  3. 2025CRADLE: Conversational RTL Design Space Exploration with LLM-based Multi-Agent Systemscited by 1
  4. 2025DeepV: A Model-Agnostic Retrieval-Augmented Framework for Verilog Code Generation with a High-Quality Knowledge Basecited by 1
  5. 2025Multimodal Chip Physical Design Engineer Assistantcited by 1
  6. 2026QiMeng-ChipV-RTL: Exploiting Information Locality for IP-level Verilog Generationcited by 1
  7. 2026Saarthi for AGI: Towards Domain-Specific General Intelligence for Formal Verificationcited by 1
  8. 2026Exploring the Agentic Frontier of Verilog Code Generationcited by 1
  9. 2026ChipMATE: Multi-Agent Training via Reinforcement Learning for Enhanced RTL Generationcited by 1
  10. 2026NoTB: Oracle-Free Triage of LLM-Generated RTL via Cross-Model Formal Consensuscited by 1
  11. 2026LLMs in Digital EDA: A perspective on shifting roles from Generation to Orchestrationcited by 1
  12. 2026Automatic Harness Evolution for Hardware Design Verification: Can LLMs Consolidate Gains Across Discovered Harnesses?cited by 1
  13. 2026Agentic AI-based Coverage Closure for Formal Verificationcited by 0
  14. 2026ChipCraftBrain: Validation-First RTL Generation via Multi-Agent Orchestrationcited by 0
  15. 2026RuC: HDL-Agnostic Rule Completion Benchmark Generationcited by 0
  16. 2026CktFormalizer: Autoformalization of Natural Language into Circuit Representationscited by 0
  17. 2026CASS-RTL: Correctness-Aware Subspace Steering for RTL Generation with LLMscited by 0
  18. 2026HierSVA: A Data Synthesis Pipeline, Dataset, and Benchmark for LLM-Driven Hierarchical Hardware Formal Verificationcited by 0
  19. 2026AUTOGATE: Automated Clock Gating via Toggling-Aware LLM-based RTL Rewritingcited by 0
  20. 2026Agentic Electronic Design Automation: A Handoff Perspectivecited by 0
  21. 2026When LLMs Over-Answer: Measuring and Mitigating Quality Issues in LLM-Based Hardware Description Language Question Answeringcited by 0
  22. 2026FinHardBench: Can LLMs Generate Latency-Aware Hardware for Financial Computing?cited by 0
  23. 2026VeriTrace: Human-Like Temporal Exploration Completes Agentic Action Spacecited by 0
  24. 2026PerfReasoning: How Well Do LLMs Reason on Hardware Performance?cited by 0
  25. 2026CovR: Coverage-Aware Hardware Verification via Reasoning-Guided Reinforcement Learningcited by 0
  26. 2026TicTacBench: Benchmarking Timing Closure Capabilities of Coding Agentscited by 0
  27. 2026ChipMEM: Verification-Grounded Memory for EDA Agentscited by 0
  28. 2026iCoder-27B: Recursive AI-Led Development of Frontier Industrial Coding Modelcited by 0
  29. 2026SpecRead: A Benchmark for Measuring Whether Language Models Understand Hardware Specificationscited by 0
  30. 2026BEHAVE: Functional Behavior Modeling Enables Self-Improving Agents for Hardware Design and Verificationcited by 0
  31. 2026VHDL-REPOBENCH: A Repository-Level Benchmark for Evaluating Large Language Models on VHDL Design Generationcited by 0
  32. 2026A Survey on LLM-Integrated Hardware Design Verificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VeriMind: Agentic LLM for Automated Verilog Generation with a Novel Evaluation Metric
  2. 2026RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Models
  3. 2026VeriGraphi: A Multi-Agent Framework of Hierarchical RTL Generation for Large Hardware Designs
  4. 2026Synthesis-in-the-Loop Evaluation of LLMs for RTL Generation: Quality, Reliability, and Failure Modes
  5. 2025VFlow: Discovering Optimal Agentic Workflows for Verilog Generation
  6. 2024EDA-Aware RTL Generation with Large Language Models
  7. 2025DUET: Agentic Design Understanding via Experimentation and Testing
  8. 2026How LLMs Fail and Generalize in RTL Coding for Hardware Design?