# HLS-Eval: A Benchmark and Framework for Evaluating LLMs on High-Level Synthesis Design Tasks

arXiv: https://arxiv.org/abs/2504.12268 · HTML: https://mlsanity.com/paper/2504.12268/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 16 Apr 2025.

## Builds on (most cited first, 15 of 15)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [VeriGen: A Large Language Model for Verilog Code Generation](https://mlsanity.com/paper/2308.00708/) (2023)
- [MG-Verilog: Multi-grained Dataset Towards Enhanced LLM-assisted Verilog Generation](https://mlsanity.com/paper/2407.01910/) (2024)
- [RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projects](https://mlsanity.com/paper/2405.17378/) (2024)
- [HLSPilot: LLM-based High-Level Synthesis](https://mlsanity.com/paper/2408.06810/) (2024)
- [Transformations of High-Level Synthesis Codes for High-Performance Computing](https://mlsanity.com/paper/1805.08288/) (2018)
- [C2HLSC: Can LLMs Bridge the Software-to-Hardware Design Gap?](https://mlsanity.com/paper/2406.09233/) (2024)
- [FlowGNN: A Dataflow Architecture for Real-Time Workload-Agnostic Graph Neural Network Inference](https://mlsanity.com/paper/2204.13103/) (2022)
- [Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis](https://mlsanity.com/paper/2502.13921/) (2025)
- [Parallel Programming for FPGAs](https://mlsanity.com/paper/1805.03648/) (2018)
- [HLSFactory: A Framework Empowering High-Level Synthesis Datasets for Machine Learning and Beyond](https://mlsanity.com/paper/2405.00820/) (2024)
- [Democratizing Domain-Specific Computing](https://mlsanity.com/paper/2209.02951/) (2022)
- [HLSDataset: Open-Source Dataset for ML-Assisted FPGA Design using High Level Synthesis](https://mlsanity.com/paper/2302.10977/) (2023)
- [GNNBuilder: An Automated Framework for Generic Graph Neural Network Accelerator Generation, Simulation, and Optimization](https://mlsanity.com/paper/2303.16459/) (2023)

## Built on by (most cited first, 10 of 10)

- [CorrectHDL: Agentic HDL Design with LLMs Leveraging High-Level Synthesis as Reference](https://mlsanity.com/paper/2511.16395/) (2025)
- [Bench4HLS: End-to-End Evaluation of LLMs in High-Level Synthesis Code Generation](https://mlsanity.com/paper/2601.19941/) (2026)
- [MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion benchmark](https://mlsanity.com/paper/2601.03708/) (2026)
- [HLS-Seek: QoR-Aware Code Generation for High-Level Synthesis via Proxy Comparative Reward Reinforcement Learning](https://mlsanity.com/paper/2605.13536/) (2026)
- [Evidence-Driven LLM Agent for C-to-Synthesizable-C Conversion and Verification](https://mlsanity.com/paper/2606.28409/) (2026)
- [From Pragmas to Partners: A Symbiotic Evolution of Agentic High-Level Synthesis](https://mlsanity.com/paper/2602.01401/) (2026)
- [ContractHIL-HLS: Contract-Aligned Multi-Agent Workflow with Hardware-in-the-Loop Feedback for HLS Design](https://mlsanity.com/paper/2607.25283/) (2026)
- [HLSmith: An Expert-Guided Agentic Framework for C/C++-to-HLS Translation](https://mlsanity.com/paper/2608.06791/) (2026)
- [PerfReasoning: How Well Do LLMs Reason on Hardware Performance?](https://mlsanity.com/paper/2609.04476/) (2026)
- [HLSFactory-Agent: Large-Scale Agentic HLS Dataset Construction from Academic and Open-Source Projects](https://mlsanity.com/paper/2609.09519/) (2026)

## Abstract (arXiv)

The rapid scaling of large language model (LLM) training and inference has driven their adoption in semiconductor design across academia and industry. While most prior work evaluates LLMs on hardware description language (HDL) tasks, particularly Verilog, designers are increasingly using high-level synthesis (HLS) to build domain-specific accelerators and complex hardware systems. However, benchmarks and tooling to comprehensively evaluate LLMs for HLS design tasks remain scarce. To address this, we introduce HLS-Eval, the first complete benchmark and evaluation framework for LLM-driven HLS design. HLS-Eval targets two core tasks: (1) generating HLS code from natural language descriptions, and (2) performing HLS-specific code edits to optimize performance and hardware efficiency. The benchmark includes 94 unique designs drawn from standard HLS benchmarks and novel sources. Each case is prepared via a semi-automated flow that produces a natural language description and a paired testbench for C-simulation and synthesis validation, ensuring each task is "LLM-ready." Beyond the benchmark, HLS-Eval offers a modular Python framework for automated, parallel evaluation of both local and hosted LLMs. It includes a parallel evaluation engine, direct HLS tool integration, and abstractions for to support different LLM interaction paradigms, enabling rapid prototyping of new benchmarks, tasks, and LLM methods. We demonstrate HLS-Eval through baseline evaluations of open-source LLMs on Vitis HLS, measuring outputs across four key metrics - parseability, compilability, runnability, and synthesizability - reflecting the iterative HLS design cycle. We also report pass@k metrics, establishing clear baselines and reusable infrastructure for the broader LLM-for-hardware community. All benchmarks, framework code, and results are open-sourced at https://github.com/stefanpie/hls-eval.