mlsanityData to 14 Oct 2026

HarmChip: Evaluating Hardware Security Centric LLM Safety via Jailbreak Benchmarking

arXiv 2604.17093 ↗First posted 18 Apr 2026cs.CRcited by 3

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

The integration of large language models (LLMs) into electronic design automation (EDA) workflows has introduced powerful capabilities for RTL generation, verification, and design optimization, but also raises critical security concerns. Malicious LLM outputs in this domain pose hardware-level threats, including hardware Trojan insertion, side-channel leakage, and intellectual property theft, that are irreversible once fabricated into silicon. Such requests often exploit semantic disguise, embedding adversarial intent within legitimate engineering language that existing safety mechanisms, trained on general-purpose hazards, fail to detect. No benchmark exists to evaluate LLM vulnerability to such domain-specific threats. We present the HarmChip benchmark to assess jailbreak susceptibility in hardware security, spanning 16 hardware security domains, 120 threats, and 360 prompts at two difficulty levels. Evaluation of state-of-the-art LLMs reveals an alignment paradox: They refuse legitimate security queries while complying with semantically disguised attacks, exposing blind spots in safety guardrails and underscoring the need for domain-aware safety alignment.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 17 of its 24 references that are in the map. First posted on arXiv 18 Apr 2026.

Builds on

17 of 24 references.

References · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  5. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  6. 2022Red Teaming Language Models with Language Modelscited by 1,026
  7. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  8. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  9. 2022Benchmarking Large Language Models for Automated Verilog RTL Code Generationcited by 115
  10. 2025A Survey of Research in Large Language Models for Electronic Design Automationcited by 41
  11. 2024Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challengescited by 32
  12. 2024LLMs and the Future of Chip Design: Unveiling Security Risks and Building Trustcited by 20
5 more
  1. 2025SALAD: Systematic Assessment of Machine Unlearning on LLM-Aided Hardware Designcited by 12
  2. 2025VeriLeaky: Navigating IP Protection vs Utility in Fine-Tuning for LLM-Driven Verilog Codingcited by 11
  3. 2025NetDeTox: Adversarial and Efficient Evasion of Hardware-Security GNNs via RL-LLM Orchestrationcited by 9
  4. 2025TrojanLoC: LLM-based Framework for RTL Trojan Localizationcited by 6
  5. 2025MetaCipher: A Time-Persistent and Universal Multi-Agent Framework for Cipher-Based Jailbreak Attacks for LLMscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026LLMs for Secure Hardware Design and Related Problems: Opportunities and Challengescited by 1
  2. 2026Hardware Design and Security in the Era of Chiplets and LLMscited by 0
  3. 2026CESBench: Benchmarking Large Language Models on Cryptographic Engineering Security for IoT Devicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SecureRAG-RTL: A Retrieval-Augmented, Multi-Agent, Zero-Shot LLM-Driven Framework for Hardware Vulnerability Detection
  2. 2025Towards LLM-based Root Cause Analysis of Hardware Design Failures
  3. 2026Semantic Consensus Decoding: Backdoor Defense for Verilog Code Generation
  4. 2026Can Agents Secure Hardware? Evaluating Agentic LLM-Driven Obfuscation for IP Protection
  5. 2025Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attacks
  6. 2025The Trojan Example: Jailbreaking LLMs through Template Filling and Unsafety Reasoning
  7. 2026Distilling Safe LLM Systems via Soft Prompts for On Device Settings
  8. 2025The Structural Safety Generalization Problem