# Bench4HLS: End-to-End Evaluation of LLMs in High-Level Synthesis Code Generation

arXiv: https://arxiv.org/abs/2601.19941 · HTML: https://mlsanity.com/paper/2601.19941/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 16 Jan 2026.

## Builds on (most cited first, 24 of 24)

- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model](https://mlsanity.com/paper/2308.05345/) (2023)
- [hls4ml: An Open-Source Codesign Workflow to Empower Scientific Low-Power Machine Learning Devices](https://mlsanity.com/paper/2103.05579/) (2021)
- [Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation](https://mlsanity.com/paper/2408.11053/) (2024)
- [Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification](https://mlsanity.com/paper/2506.14074/) (2025)
- [A Survey of Research in Large Language Models for Electronic Design Automation](https://mlsanity.com/paper/2501.09655/) (2025)
- [HLSPilot: LLM-based High-Level Synthesis](https://mlsanity.com/paper/2408.06810/) (2024)
- [Automated C/C++ Program Repair for High-Level Synthesis via Large Language Models](https://mlsanity.com/paper/2407.03889/) (2024)
- [RTL++: Graph-enhanced LLM for RTL Code Generation](https://mlsanity.com/paper/2505.13479/) (2025)
- [HDLCoRe: A Training-Free Framework for Mitigating Hallucinations in LLM-Generated HDL](https://mlsanity.com/paper/2503.16528/) (2025)
- [C2HLSC: Can LLMs Bridge the Software-to-Hardware Design Gap?](https://mlsanity.com/paper/2406.09233/) (2024)
- [Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis](https://mlsanity.com/paper/2502.13921/) (2025)
- [HLS-Eval: A Benchmark and Framework for Evaluating LLMs on High-Level Synthesis Design Tasks](https://mlsanity.com/paper/2504.12268/) (2025)
- [Parallel Programming for FPGAs](https://mlsanity.com/paper/1805.03648/) (2018)
- [Can Reasoning Models Reason about Hardware? An Agentic HLS Perspective](https://mlsanity.com/paper/2503.12721/) (2025)
- [SAGE-HLS: Syntax-Aware AST-Guided LLM for High-Level Synthesis Code Generation](https://mlsanity.com/paper/2508.03558/) (2025)
- [LIFT: LLM-Based Pragma Insertion for HLS via GNN Supervised Fine-Tuning](https://mlsanity.com/paper/2504.21187/) (2025)
- [SynthAI: A Multi Agent Generative AI Framework for Automated Modular HLS Design Generation](https://mlsanity.com/paper/2405.16072/) (2024)
- [Are LLMs Any Good for High-Level Synthesis?](https://mlsanity.com/paper/2408.10428/) (2024)
- [Optimizing High-Level Synthesis Designs with Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2410.07356/) (2024)
- [DecoRTL: A Run-time Decoding Framework for RTL Code Generation with LLMs](https://mlsanity.com/paper/2507.02226/) (2025)
- [TimelyHLS: LLM-Based Timing-Aware and Architecture-Specific FPGA HLS Optimization](https://mlsanity.com/paper/2507.17962/) (2025)
- [Agentic-HLS: An agentic reasoning based high-level synthesis system using large language models (AI for EDA workshop 2024)](https://mlsanity.com/paper/2412.01604/) (2024)

## Built on by (most cited first, 2 of 2)

- [Exploring the Agentic Frontier of Verilog Code Generation](https://mlsanity.com/paper/2603.19347/) (2026)
- [PerfReasoning: How Well Do LLMs Reason on Hardware Performance?](https://mlsanity.com/paper/2609.04476/) (2026)

## Abstract (arXiv)

In last two years, large language models (LLMs) have shown strong capabilities in code generation, including hardware design at register-transfer level (RTL). While their use in high-level synthesis (HLS) remains comparatively less mature, the ratio of HLS- to RTL-focused studies has shifted from 1:10 to 2:10 in the past six months, indicating growing interest in leveraging LLMs for high-level design entry while relying on downstream synthesis for optimization. This growing trend highlights the need for a comprehensive benchmarking and evaluation framework dedicated to LLM-based HLS. To address this, We present Bench4HLS for evaluating LLM-generated HLS designs. Bench4HLS comprises 170 manually drafted and validated case studies, spanning small kernels to complex accelerators, curated from widely used public repositories. The framework supports fully automated assessment of compilation success, functional correctness via simulation, and synthesis feasibility/optimization. Crucially, Bench4HLS integrates a pluggable API for power, performance, and area (PPA) analysis across various HLS toolchains and architectures, demonstrated here with Xilinx Vitis HLS and validated on Catapult HLS. By providing a structured, extensible, and plug-and-play testbed, Bench4HLS establishes a foundational methodology for benchmarking LLMs in HLS workflows.