# ProtocolLLM: RTL Benchmark for SystemVerilog Generation of Communication Protocols

arXiv: https://arxiv.org/abs/2506.07945 · HTML: https://mlsanity.com/paper/2506.07945/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Builds on (most cited first, 23 of 23)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [VeriGen: A Large Language Model for Verilog Code Generation](https://mlsanity.com/paper/2308.00708/) (2023)
- [RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model](https://mlsanity.com/paper/2308.05345/) (2023)
- [OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models](https://mlsanity.com/paper/2411.04905/) (2024)
- [Benchmarking Large Language Models for Automated Verilog RTL Code Generation](https://mlsanity.com/paper/2212.11140/) (2022)
- [ChipGPT: How far are we from natural language hardware design](https://mlsanity.com/paper/2305.14019/) (2023)
- [BetterV: Controlled Verilog Generation with Discriminative Guidance](https://mlsanity.com/paper/2402.03375/) (2024)
- [A Survey of Research in Large Language Models for Electronic Design Automation](https://mlsanity.com/paper/2501.09655/) (2025)
- [Towards LLM-Powered Verilog RTL Assistant: Self-Verification and Self-Correction](https://mlsanity.com/paper/2406.00115/) (2024)
- [AssertLLM: Generating and Evaluating Hardware Verification Assertions from Design Specifications via Multi-LLMs](https://mlsanity.com/paper/2402.00386/) (2024)
- [Exploring and Characterizing Large Language Models For Embedded System Development and Debugging](https://mlsanity.com/paper/2307.03817/) (2023)
- [VFlow: Discovering Optimal Agentic Workflows for Verilog Generation](https://mlsanity.com/paper/2504.03723/) (2025)
- [VHDL-Eval: A Framework for Evaluating Large Language Models in VHDL Code Generation](https://mlsanity.com/paper/2406.04379/) (2024)
- [VeriThoughts: Enabling Automated Verilog Code Generation using Reasoning and Formal Verification](https://mlsanity.com/paper/2505.20302/) (2025)
- [All Artificial, Less Intelligence: GenAI through the Lens of Formal Verification](https://mlsanity.com/paper/2403.16750/) (2024)
- [TuRTLe: A Unified Evaluation of LLMs for RTL Generation](https://mlsanity.com/paper/2504.01986/) (2025)
- [Evaluating Large Language Models for Automatic Register Transfer Logic Generation via High-Level Synthesis](https://mlsanity.com/paper/2408.02793/) (2024)

## Built on by (most cited first, 1 of 1)

- [LLM-Empowered Agentic MAC Protocols: A Dynamic Stackelberg Game Approach](https://mlsanity.com/paper/2510.10895/) (2025)

## Abstract (arXiv)

Recent advances in large language models (LLMs) have demonstrated strong performance in generating code for general-purpose programming languages. However, their potential for hardware description languages (HDLs), such as SystemVerilog, remains largely unexplored. HDL code generation poses unique challenges due to strict timing semantics, concurrency, and synthesizability constraints essential for correct hardware functionality. Further, HDL-based design flows encompass a broad set of tasks beyond structural code generation, including testbench development, assertion-based verification, timing closure, and protocol-level integration for on-chip communication. In this work, we evaluate the capabilities of both open-source and state-of-the-art LLMs in generating synthesizable and functionally accurate SystemVerilog implementations of widely used communication protocols that are critical components of embedded and System-on-Chip (SoC) systems. We introduce ProtocolLLM, the first benchmark suite specifically targeting these protocols with tasks spanning multiple design abstraction levels and varying prompt specificity. Our evaluation method also focuses on timing correctness in addition to synthesizability and syntactic correctness. We observe that most of the models fail to generate SystemVerilog code for communication protocols that follow timing constrains.