mlsanityData to 14 Oct 2026

RealBench: Benchmarking Verilog Generation Models with Real-World IP Designs

arXiv 2507.16200 ↗First posted 22 Jul 2025cs.LG cs.ARcited by 14

Not on the map. Why?

Abstract (arXiv)

The automatic generation of Verilog code using Large Language Models (LLMs) has garnered significant interest in hardware design automation. However, existing benchmarks for evaluating LLMs in Verilog generation fall short in replicating real-world design workflows due to their designs' simplicity, inadequate design specifications, and less rigorous verification environments. To address these limitations, we present RealBench, the first benchmark aiming at real-world IP-level Verilog generation tasks. RealBench features complex, structured, real-world open-source IP designs, multi-modal and formatted design specifications, and rigorous verification environments, including 100% line coverage testbenches and a formal checker. It supports both module-level and system-level tasks, enabling comprehensive assessments of LLM capabilities. Evaluations on various LLMs and agents reveal that even one of the best-performing LLMs, o1-preview, achieves only a 13.3% pass@1 on module-level tasks and 0% on system-level tasks, highlighting the need for stronger Verilog generation models in the future. The benchmark is open-sourced at https://github.com/IPRC-DIP/RealBench.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 22 Jul 2025.

Builds on

33 of 45 references.

References · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  8. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 252
  9. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 174
  10. 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 148
  11. 2023Chip-Chat: Challenges and Opportunities in Conversational Hardware Designcited by 140
  12. 2023ChatEDA: A Large Language Model Powered Autonomous Agent for EDAcited by 122
21 more
  1. 2022Benchmarking Large Language Models for Automated Verilog RTL Code Generationcited by 115
  2. 2023ChipGPT: How far are we from natural language hardware designcited by 105
  3. 2023RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Modelscited by 100
  4. 2024BetterV: Controlled Verilog Generation with Discriminative Guidancecited by 100
  5. 2024VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Toolcited by 85
  6. 2023RTLCoder: Outperforming GPT-3.5 in Design RTL Generation with Our Open-Source Dataset and Lightweight Solutioncited by 71
  7. 2023GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Modelscited by 61
  8. 2024OriGen:Enhancing RTL Code Generation with Code-to-Code Augmentation and Self-Reflectioncited by 56
  9. 2024MG-Verilog: Multi-grained Dataset Towards Enhanced LLM-assisted Verilog Generationcited by 46
  10. 2024CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repaircited by 42
  11. 2024Data is all you need: Finetuning LLMs for Chip Design via an Automated design-data augmentation frameworkcited by 36
  12. 2024AutoVCoder: A Systematic Framework for Automated Verilog Code Generation using LLMscited by 36
  13. 2024AssertLLM: Generating and Evaluating Hardware Verification Assertions from Design Specifications via Multi-LLMscited by 31
  14. 2024RTLRewriter: Methodologies for Large Models aided RTL Code Optimizationcited by 31
  15. 2025OpenLLM-RTL: Open Dataset and Benchmark for LLM-Aided Design RTL Generationcited by 31
  16. 2024HDLdebugger: Streamlining HDL debugging with Large Language Modelscited by 26
  17. 2024RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projectscited by 26
  18. 2024AIvril: AI-Driven RTL Generation With Verification In-The-Loopcited by 15
  19. 2023On the Viability of using LLMs for SW/HW Co-Design: An Example in Designing CiM DNN Acceleratorscited by 11
  20. 2024Natural language is not enough: Benchmarking multi-modal generative AI for Verilog generationcited by 11
  21. 2024SpecLLM: Exploring Generation and Review of VLSI Design Specification with Large Language Modelcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 13.

14 papers.

Sort:

By area:

  1. 2025Large Language Model for Verilog Code Generation: Literature Review and the Road Aheadcited by 8
  2. 2026ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Designcited by 8
  3. 2026InCoder-32B-Thinking: Industrial Code World Model for Thinkingcited by 4
  4. 2026Advances in GRPO for Generation Models: A Surveycited by 2
  5. 2026HAVEN: Hybrid Automated Verification ENgine for UVM Testbench Synthesis with LLMscited by 2
  6. 2026Agentic Agile-V: From Vibe Coding to Verified Engineering in Software and Hardware Developmentcited by 2
  7. 2026QiMeng-ChipV-RTL: Exploiting Information Locality for IP-level Verilog Generationcited by 1
  8. 2026LLMs in Digital EDA: A perspective on shifting roles from Generation to Orchestrationcited by 1
  9. 2026Agentic Electronic Design Automation: A Handoff Perspectivecited by 0
  10. 2026BC-Bench: Evaluating Agentic Engineering in a Domain-Specific Language for ERPcited by 0
  11. 2026iCoder-27B: Recursive AI-Led Development of Frontier Industrial Coding Modelcited by 0
  12. 2026CodeSkill: Latent Skill Abstraction for Long-Horizon Code Agentscited by 0
2 more
  1. 2026BEHAVE: Functional Behavior Modeling Enables Self-Improving Agents for Hardware Design and Verificationcited by 0
  2. 2026From Dead Code and Static Requirements to Working Engines: Software Revival with Coding Agentscited by 0