mlsanityData to 14 Oct 2026

Configuration Over Selection: Hyperparameter Sensitivity Exceeds Model Differences in Open-Source LLMs for RTL Generation

arXiv 2604.17102 ↗First posted 18 Apr 2026cs.AR cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Benchmarking of open-source LLMs for hardware design focuses on which LLMs to use, while treating inference-time decoding configuration as a secondary concern. This work shows that it matters more how an LLM is configured than which model is selected. Benchmarking 26 open-source LLMs on VerilogEval and RTLLM with synthesis-in-the-loop evaluation, the study first maps the current capability landscape and then conducts an extensive 108-configuration hyperparameter sweep on three prominent models. The sweep reveals absolute pass-rate gaps of up to 25.5% between the best and worst settings for the same LLM, which is 5x larger than the average spread observed across various model families under their respective default configurations. Ranking all configurations by Spearman's ρ across the two benchmark suites yields near-zero correlation, demonstrating that optimal configurations do not transfer. These results show that benchmarking conducted under default hyperparameters confounds model capabilities with configuration effects. Realizing the full potential of open-source LLMs for RTL generation requires architecture and benchmark aware hyperparameter selection, as enabled by the proposed methodology.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 17 of its 21 references that are in the map. First posted on arXiv 18 Apr 2026.

Builds on

17 of 21 references.

References · 17

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 252
  4. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 200
  5. 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 148
  6. 2022Benchmarking Large Language Models for Automated Verilog RTL Code Generationcited by 115
  7. 2024OriGen:Enhancing RTL Code Generation with Code-to-Code Augmentation and Self-Reflectioncited by 56
  8. 2024Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generationcited by 45
  9. 2023Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Modelscited by 38
  10. 2024Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challengescited by 32
  11. 2025OpenLLM-RTL: Open Dataset and Benchmark for LLM-Aided Design RTL Generationcited by 31
  12. 2024LLMs and the Future of Chip Design: Unveiling Security Risks and Building Trustcited by 20
5 more
  1. 2025VeriLeaky: Navigating IP Protection vs Utility in Fine-Tuning for LLM-Driven Verilog Codingcited by 11
  2. 2025TuRTLe: A Unified Evaluation of LLMs for RTL Generationcited by 9
  3. 2025NetDeTox: Adversarial and Efficient Evasion of Hardware-Security GNNs via RL-LLM Orchestrationcited by 9
  4. 2026Synthesis-in-the-Loop Evaluation of LLMs for RTL Generation: Quality, Reliability, and Failure Modescited by 4
  5. 2025Pluto: A Benchmark for Evaluating Efficiency of LLM-generated Hardware Codecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026LLMs for Secure Hardware Design and Related Problems: Opportunities and Challengescited by 1