mlsanityData to 14 Oct 2026

RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projects

arXiv 2405.17378 ↗First posted 27 May 2024cs.LG cs.ARcited by 26

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated potential in assisting with Register Transfer Level (RTL) design tasks. Nevertheless, there remains to be a significant gap in benchmarks that accurately reflect the complexity of real-world RTL projects. To address this, this paper presents RTL-Repo, a benchmark specifically designed to evaluate LLMs on large-scale RTL design projects. RTL-Repo includes a comprehensive dataset of more than 4000 Verilog code samples extracted from public GitHub repositories, with each sample providing the full context of the corresponding repository. We evaluate several state-of-the-art models on the RTL-Repo benchmark, including GPT-4, GPT-3.5, Starcoder2, alongside Verilog-specific models like VeriGen and RTLCoder, and compare their performance in generating Verilog code for complex projects. The RTL-Repo benchmark provides a valuable resource for the hardware design community to assess and compare LLMs' performance in real-world RTL design scenarios and train LLMs specifically for Verilog code generation in complex, multi-file RTL projects. RTL-Repo is open-source and publicly available on Github.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 13 of its 17 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

13 of 17 references.

References · 13

  1. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  2. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  3. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  4. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  5. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  6. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 252
  7. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 247
  8. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 200
  9. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 183
  10. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 174
  11. 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 148
  12. 2024Make Your LLM Fully Utilize the Contextcited by 119
1 more
  1. 2022Benchmarking Large Language Models for Automated Verilog RTL Code Generationcited by 115

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 15; 2026: 9.

26 papers.

Sort:

By area:

  1. 2024Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generationcited by 45
  2. 2025DeepCircuitX: A Comprehensive Repository-Level Dataset for RTL Code Understanding, Generation, and PPA Analysiscited by 21
  3. 2025VFlow: Discovering Optimal Agentic Workflows for Verilog Generationcited by 19
  4. 2025RealBench: Benchmarking Verilog Generation Models with Real-World IP Designscited by 14
  5. 2025ReasoningV: Efficient Verilog Code Generation with Adaptive Hybrid Reasoning Modelcited by 12
  6. 2025SALAD: Systematic Assessment of Machine Unlearning on LLM-Aided Hardware Designcited by 12
  7. 2026ACE-RTL: When Agentic Context Evolution Meets RTL-Specialized LLMscited by 12
  8. 2025Enhancing Large Language Models for Hardware Verification: A Novel SystemVerilog Assertion Datasetcited by 10
  9. 2025HLS-Eval: A Benchmark and Framework for Evaluating LLMs on High-Level Synthesis Design Taskscited by 10
  10. 2025TuRTLe: A Unified Evaluation of LLMs for RTL Generationcited by 9
  11. 2025VerilogDB: The Largest, Highest-Quality Dataset with a Preprocessing Framework for LLM-based RTL Generationcited by 8
  12. 2025Large Language Model for Verilog Code Generation: Literature Review and the Road Aheadcited by 8
14 more
  1. 2025RTLRepoCoder: Repository-Level RTL Code Completion through the Combination of Fine-Tuning and Retrieval Augmentationcited by 7
  2. 2024Are LLMs Any Good for High-Level Synthesis?cited by 6
  3. 2026Dr. RTL: Autonomous Agentic RTL Optimization through Tool-Grounded Self-Improvementcited by 6
  4. 2025ASIC-Agent: An Autonomous Multi-Agent System for ASIC Design with Benchmark Evaluationcited by 2
  5. 2025Towards Optimal Circuit Generation: Multi-Agent Collaboration Meets Collective Intelligencecited by 1
  6. 2025HDLxGraph: Bridging Large Language Models and HDL Repositories via HDL Graph Databasescited by 1
  7. 2026MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion benchmarkcited by 1
  8. 2026RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Modelscited by 1
  9. 2025Lorecast: Layout-Aware Performance and Power Forecasting from Natural Languagecited by 0
  10. 2026RuC: HDL-Agnostic Rule Completion Benchmark Generationcited by 0
  11. 2026Agentic Electronic Design Automation: A Handoff Perspectivecited by 0
  12. 2026VeriTrace: Human-Like Temporal Exploration Completes Agentic Action Spacecited by 0
  13. 2026TicTacBench: Benchmarking Timing Closure Capabilities of Coding Agentscited by 0
  14. 2026VHDL-REPOBENCH: A Repository-Level Benchmark for Evaluating Large Language Models on VHDL Design Generationcited by 0