mlsanityData to 14 Oct 2026

Code as a Weapon: A Consensus-Labeled Prompt Bank for Measuring Coding-Model Compliance with Malicious-Code Requests

arXiv 2605.28734 ↗First posted 27 May 2026cs.CR cs.CL cs.LGcited by 0

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

A general-purpose language model that answers a harmful question returns text; a coding model that complies with a malicious request can return a working weapon: a keylogger, ransomware, an exploit that runs as written. This asymmetry in the severity of a single act of compliance implies coding-specialized models should clear a higher refusal bar than general-purpose chat models, not a lower one, yet the field cannot tell whether they do. Refusal benchmarks for malicious code are fragmented: they mix requests for executable software with requests for harmful security knowledge and report refusal rates over non-comparable corpora. This paper's central result is that the CODE-versus-KNOWLEDGE classification axis established in a prior four-corpus release remains stable under a substantially expanded corpus pool and an independently refreshed judge panel, evidence that it measures a real construct rather than an artifact of the prompts or judges. Eight corpora spanning diverse elicitation paradigms (direct, jailbreak-decorated, indirect, and agent/interpreter: ASTRA, CySecBench, AdvBench/harmful_behaviors, JailbreakBench, MalwareBench, RedCode, RMCBench, Scam2Prompt) are classified under a five-judge consensus protocol (6,675 prompts x 5 judges = 33,375 calls), reaching Fleiss' kappa = 0.767 [95% CI 0.755, 0.777] ("substantial"). Critically, the panel shares no judge with the prior release (five paid commercial APIs replaced by five open-weight models from five vendors), yet the two panels agree on 94.45% of the 3,133 shared prompts and reach Cohen's kappa = 0.952 [0.942, 0.963] on the 3,031-prompt binary overlap: the axis survives near-total panel replacement. The released bank comprises 4,748 consensus-CODE and 1,923 consensus-KNOWLEDGE prompts, a reliability-quantified benchmark whose central classification axis is shown stable across corpus expansion and judge-panel replacement.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 14 of its 20 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

14 of 20 references.

References · 14

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  2. 2024A Survey on LLM-as-a-Judgecited by 1,423
  3. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  4. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  5. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agentscited by 90
  6. 2026Qwen3-Coder-Next Technical Reportcited by 71
  7. 2024RMCBench: Benchmarking Large Language Models' Resistance to Malicious Codecited by 28
  8. 2024Annotation alignment: Comparing LLM and human annotations of conversational safetycited by 23
  9. 2025CySecBench: Generative AI-based CyberSecurity-focused Prompt Dataset for Benchmarking Large Language Modelscited by 10
  10. 2025LLMs Caught in the Crossfire: Malware Requests and Jailbreak Challengescited by 9
  11. 2025ASTRA: Autonomous Spatial-Temporal Red-teaming for AI Software Assistantscited by 6
  12. 2025Scam2Prompt: A Scalable Framework for Auditing Malicious Scam Endpoints in Production LLMscited by 2
2 more
  1. 2026A Validated Prompt Bank for Malicious Code Generation: Separating Executable Weapons from Security Knowledge in 1,554 Consensus-Labeled Promptscited by 2
  2. 2026Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025)cited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal
  2. 2026SpecCloak: Probing Harmful Code Generation through Engineering Specifications
  3. 2026Same Request, Different Boundary: Evaluating Cybersecurity Assistance across Conversational Contexts
  4. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models
  5. 2026Beyond Refusal: A Same-Lineage Study of Aligned and Abliterated LLMs for Vulnerability Analysis
  6. 2025Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks
  7. 2026Refused in Chat, Written in Code: Workflow-Level Jailbreak Construction in IDE Coding Agents
  8. 2025Understanding and Mitigating Over-refusal for Large Language Models via Representation Intervention