Same Request, Different Boundary: Evaluating Cybersecurity Assistance across Conversational Contexts
Not on the map. Why?
Abstract (arXiv)
Cybersecurity assistance is dual use, requiring decisions about when to answer, limit, or refuse a request. Prior work studies refusal evaluation and conversational jailbreaking, but how conversational histories shift assistance boundaries in both directions for identical cybersecurity requests remains underexplored. We introduce 3R-Bench, a benchmark of 150 real-world cybersecurity requests evaluated across eight LLMs under direct prompting and two predefined conversational settings. We separately code refusal, bounded assistance, compliance, and provider blocking to characterize delivered assistance. On 400 matched ambiguous request-model pairs, compliance is 75.3% directly, 57.8% after refused precedent, and 81.0% after accepted precedent. A smaller repeated-generation panel reproduces the refusal-acceptance gap. On a balanced 75-request panel reusing the direct baseline, compliance decreases from 74.2% directly to 57.2% after neutral task-decomposition dialogue and 60.3% after failure-ended dialogue. These findings show that cybersecurity assistance varies with conversational history and model deployment, motivating evaluation beyond direct-request refusal rates.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 7 of its 9 references that are in the map. First posted on arXiv 1 Sep 2026.
Builds on
7 of 9 references.
References · 7
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
- 2024A StrongREJECT for Empty Jailbreakscited by 385
- 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
- 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 89
- 2025CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scalecited by 78
- 2026ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?cited by 29
- 2026Defensive Refusal Bias: How Safety Alignment Fails Cyber Defenderscited by 10
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026A Content-Based Framework for Cybersecurity Refusal Decisions in Large Language Models
- 2026Code as a Weapon: A Consensus-Labeled Prompt Bank for Measuring Coding-Model Compliance with Malicious-Code Requests
- 2026Ablating Safety: Mechanisms for Removing Alignment in Language Models for Security Applications
- 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning
- 2024The Art of Saying No: Contextual Noncompliance in Language Models
- 2026Do Language Models Know When They'll Refuse? Probing Introspective Awareness of Safety Boundaries
- 2026A New Framework for Cybersecurity Refusals in AI Agents
- 2024Defending Against Social Engineering Attacks in the Age of LLMs