mlsanityData to 7 Oct 2026

Verdict Without the Rule: Diagnosing and Auditing Regulatory Rule Sensitivity in LLM Compliance Systems

arXiv 2610.12313 ↗First posted 8 Oct 2026cs.AI cs.CL cs.CYcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language model compliance systems are deployed on the assumption that a verdict depends on the regulatory rule it is given. We test this directly across five models and 20 regulatory and platform-policy domains: delete, swap, or negate the governing rule while holding the case fixed, and check whether the verdict changes (OCS) or the model's internal representation of compliance shifts at all (ICS-delta). Neither moves much: models' verdicts are often invariant to substantial perturbations of the supplied rule, and the guard model, evaluated here under a custom-rule adaptation of its native taxonomy, is the least rule-sensitive and least accurate of the five, barely above chance (51%, versus 90-92% for general-purpose models). This reflects easy cases more than blanket neglect: on cases where deleting the rule changes a previously correct model prediction, models do track it closely. Neither better prompting nor direct intervention on the model's internal representations closes this gap. Accuracy alone does not establish that a compliance verdict is grounded in the supplied rule.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 29 references that are in the map. First posted on arXiv 8 Oct 2026.

Builds on

14 of 29 references.

References · 14

  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  2. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  3. 2025Qwen3Guard Technical Reportcited by 125
  4. 2025GraphCompliance: Aligning Policy and Context Graphs for LLM-Based Regulatory Compliancecited by 9
  5. 2024LegiLM: A Fine-Tuned Legal Language Model for Data Compliancecited by 6
  6. 2025Training-Free Policy Violation Detection via Activation-Space Whitening in LLMscited by 6
  7. 2026LPG: Balancing Efficiency and Policy Reasoning in Latent Policy Guardrailscited by 4
  8. 2026CourtGuard: A Model-Agnostic Framework for Zero-Shot Policy Adaptation in LLM Safetycited by 2
  9. 2026ComplianceNLP: Knowledge-Graph-Augmented RAG for Multi-Framework Regulatory Gap Detectioncited by 2
  10. 2026FinGuard: Detecting Financial Regulatory Non-Compliance in LLM Interactionscited by 2
  11. 2026RuleSafe-VL: Evaluating Rule-Conditioned Decision Reasoning in Vision-Language Content Moderationcited by 1
  12. 2026Citation-Closure Retrieval and Per-Rule Attribution for Real-World Regulatory Compliance Question Answeringcited by 1
2 more
  1. 2026The Refusal Residue: When Probes Catch Alignment Faking and When They Don'tcited by 1
  2. 2026What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Modelscited by 1

Built on by

Not cited yet.