ProtocolLLM: RTL Benchmark for SystemVerilog Generation of Communication Protocols
Not on the map. Why?
Abstract (arXiv)
Recent advances in large language models (LLMs) have demonstrated strong performance in generating code for general-purpose programming languages. However, their potential for hardware description languages (HDLs), such as SystemVerilog, remains largely unexplored. HDL code generation poses unique challenges due to strict timing semantics, concurrency, and synthesizability constraints essential for correct hardware functionality. Further, HDL-based design flows encompass a broad set of tasks beyond structural code generation, including testbench development, assertion-based verification, timing closure, and protocol-level integration for on-chip communication. In this work, we evaluate the capabilities of both open-source and state-of-the-art LLMs in generating synthesizable and functionally accurate SystemVerilog implementations of widely used communication protocols that are critical components of embedded and System-on-Chip (SoC) systems. We introduce ProtocolLLM, the first benchmark suite specifically targeting these protocols with tasks spanning multiple design abstraction levels and varying prompt specificity. Our evaluation method also focuses on timing correctness in addition to synthesizability and syntactic correctness. We observe that most of the models fail to generate SystemVerilog code for communication protocols that follow timing constrains.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 9 Jun 2025.
Builds on
23 of 36 references.
References · 23
- 2023GPT-4 Technical Reportcited by 18,396
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2024Qwen2.5-Coder Technical Reportcited by 1,203
- 2023StarCoder: may the source be with you!cited by 799
- 2024A Survey on Large Language Models for Code Generationcited by 658
- 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
- 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 252
- 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 200
- 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 148
- 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 130
- 2022Benchmarking Large Language Models for Automated Verilog RTL Code Generationcited by 115
- 2023ChipGPT: How far are we from natural language hardware designcited by 105
11 more
- 2024BetterV: Controlled Verilog Generation with Discriminative Guidancecited by 100
- 2025A Survey of Research in Large Language Models for Electronic Design Automationcited by 41
- 2024Towards LLM-Powered Verilog RTL Assistant: Self-Verification and Self-Correctioncited by 39
- 2024AssertLLM: Generating and Evaluating Hardware Verification Assertions from Design Specifications via Multi-LLMscited by 31
- 2023Exploring and Characterizing Large Language Models For Embedded System Development and Debuggingcited by 20
- 2025VFlow: Discovering Optimal Agentic Workflows for Verilog Generationcited by 19
- 2024VHDL-Eval: A Framework for Evaluating Large Language Models in VHDL Code Generationcited by 15
- 2025VeriThoughts: Enabling Automated Verilog Code Generation using Reasoning and Formal Verificationcited by 15
- 2024All Artificial, Less Intelligence: GenAI through the Lens of Formal Verificationcited by 11
- 2025TuRTLe: A Unified Evaluation of LLMs for RTL Generationcited by 9
- 2024Evaluating Large Language Models for Automatic Register Transfer Logic Generation via High-Level Synthesiscited by 5
Built on by
Citing papers per year: 2025: 1.
1 paper.