mlsanityData to 14 Oct 2026

HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generation

arXiv 2601.13864 ↗First posted 20 Jan 2026cs.CR cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are increasingly used for hardware and firmware code generation, but existing studies primarily evaluate functional correctness while largely overlooking security. However, LLM-generated code that appears functionally sound may embed security flaws which could induce catastrophic damages after deployment. This critical research gap motivates us to design a benchmark for assessing security awareness under realistic specifications. In this work, we introduce HardSecBench, a benchmark with 924 tasks spanning Verilog Register Transfer Level (RTL) and firmware-level C, covering 76 hardware-relevant Common Weakness Enumeration (CWE) entries. Each task includes a structured specification, a secure reference implementation, and executable tests. To automate artifact synthesis, we propose a multi-agent pipeline that decouples synthesis from verification and grounds evaluation in execution evidence, enabling reliable evaluation. We evaluate diverse LLMs and find that they often satisfy functional requirements while leaving security risks. We also find that security results vary with prompting. These findings highlight pressing challenges and offer actionable insights for future advancements in LLM-assisted hardware design. Our data and code are available at https://github.com/chenqirui2002/HardSecBench.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 26 of its 42 references that are in the map. First posted on arXiv 20 Jan 2026.

Builds on

26 of 42 references.

References · 26

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  2. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 252
  3. 2022Do Users Write More Insecure Code with AI Assistants?cited by 192
  4. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 174
  5. 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 148
  6. 2023Chip-Chat: Challenges and Opportunities in Conversational Hardware Designcited by 140
  7. 2023ChatEDA: A Large Language Model Powered Autonomous Agent for EDAcited by 122
  8. 2022Benchmarking Large Language Models for Automated Verilog RTL Code Generationcited by 115
  9. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 96
  10. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 89
  11. 2025CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generationcited by 59
  12. 2024Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generationcited by 45
14 more
  1. 2024AutoBench: Automatic Testbench Generation and Evaluation Using LLMs for HDL Designcited by 41
  2. 2024Constrained Decoding for Secure Code Generationcited by 37
  3. 2025QiMeng-CodeV-R1: Reasoning-Enhanced Verilog Generationcited by 35
  4. 2024AssertLLM: Generating Hardware Verification Assertions from Design Specifications via Multi-LLMscited by 31
  5. 2025SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Codecited by 19
  6. 2025Spec2RTL-Agent: Automated Hardware Code Generation from Complex Specifications Using LLM Agent Systemscited by 18
  7. 2025HDLCoRe: A Training-Free Framework for Mitigating Hallucinations in LLM-Generated HDLcited by 17
  8. 2024AssertionBench: A Benchmark to Evaluate Large-Language Models for Assertion Generationcited by 16
  9. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 14
  10. 2021AutoSVA: Democratizing Formal Verification of RTL Module Interactionscited by 12
  11. 2025RESCUE: Retrieval Augmented Secure Code Generationcited by 7
  12. 2025CircuitGuard: Mitigating LLM Memorization in RTL Code Generation Against IP Leakagecited by 3
  13. 2025SecFSM: Knowledge Graph-Guided Verilog Code Generation for Secure Finite State Machines in Systems-on-Chipcited by 2
  14. 2025VerilogLAVD: LLM-Aided Rule Generation for Vulnerability Detection in Verilogcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026EduPluginBench: Executable Assurance for AI-Generated Educational Pluginscited by 0
  2. 2026Adversarial Testing of Automated Program Repair Agents for Security Vulnerabilitiescited by 0
  3. 2026Trust, but Validate the Instrument: Auditing AI-Generated RTL Verification Plans on Authored Security-Regression Proxiescited by 0
  4. 2026CESBench: Benchmarking Large Language Models on Cryptographic Engineering Security for IoT Devicescited by 0
  5. 2026Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluationcited by 0