Verdict Without the Rule: Diagnosing and Auditing Regulatory Rule Sensitivity in LLM Compliance Systems
Not on the map. Why?
Abstract (arXiv)
Large language model compliance systems are deployed on the assumption that a verdict depends on the regulatory rule it is given. We test this directly across five models and 20 regulatory and platform-policy domains: delete, swap, or negate the governing rule while holding the case fixed, and check whether the verdict changes (OCS) or the model's internal representation of compliance shifts at all (ICS-delta). Neither moves much: models' verdicts are often invariant to substantial perturbations of the supplied rule, and the guard model, evaluated here under a custom-rule adaptation of its native taxonomy, is the least rule-sensitive and least accurate of the five, barely above chance (51%, versus 90-92% for general-purpose models). This reflects easy cases more than blanket neglect: on cases where deleting the rule changes a previously correct model prediction, models do track it closely. Neither better prompting nor direct intervention on the model's internal representations closes this gap. Accuracy alone does not establish that a compliance verdict is grounded in the supplied rule.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 29 references that are in the map. First posted on arXiv 8 Oct 2026.
Builds on
14 of 29 references.
References · 14
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
- 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
- 2025Qwen3Guard Technical Reportcited by 125
- 2025GraphCompliance: Aligning Policy and Context Graphs for LLM-Based Regulatory Compliancecited by 9
- 2024LegiLM: A Fine-Tuned Legal Language Model for Data Compliancecited by 6
- 2025Training-Free Policy Violation Detection via Activation-Space Whitening in LLMscited by 6
- 2026LPG: Balancing Efficiency and Policy Reasoning in Latent Policy Guardrailscited by 4
- 2026CourtGuard: A Model-Agnostic Framework for Zero-Shot Policy Adaptation in LLM Safetycited by 2
- 2026ComplianceNLP: Knowledge-Graph-Augmented RAG for Multi-Framework Regulatory Gap Detectioncited by 2
- 2026FinGuard: Detecting Financial Regulatory Non-Compliance in LLM Interactionscited by 2
- 2026RuleSafe-VL: Evaluating Rule-Conditioned Decision Reasoning in Vision-Language Content Moderationcited by 1
- 2026Citation-Closure Retrieval and Per-Rule Attribution for Real-World Regulatory Compliance Question Answeringcited by 1
Built on by
Not cited yet.