COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMs
Not on the map. Why?
Abstract (arXiv)
As large language models are deployed in high-stakes enterprise applications, from healthcare to finance, ensuring adherence to organization-specific policies has become essential. Yet existing safety evaluations focus exclusively on universal harms. We present COMPASS (Company/Organization Policy Alignment Assessment), the first systematic framework for evaluating whether LLMs comply with organizational allowlist and denylist policies. We apply COMPASS to eight diverse industry scenarios, generating and validating 5,920 queries that test both routine compliance and adversarial robustness through strategically designed edge cases. Evaluating seven state-of-the-art models, we uncover a fundamental asymmetry: models reliably handle legitimate requests (>95% accuracy) but catastrophically fail at enforcing prohibitions, refusing only 13-40% of adversarial denylist violations. These results demonstrate that current LLMs lack the robustness required for policy-critical deployments, establishing COMPASS as an essential evaluation framework for organizational AI safety.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 15 of its 89 references that are in the map. First posted on arXiv 5 Jan 2026.
Builds on
15 of 89 references.
References · 15
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 259
- 2024A Complete Survey on LLM-based AI Chatbotscited by 71
- 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 36
- 2024The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approachescited by 18
- 2025Safety Through Reasoning: An Empirical Study of Reasoning Guardrail Modelscited by 16
- 2025from Benign import Toxic: Jailbreaking the Language Model via Adversarial Metaphorscited by 15
- 2024You Know What I'm Saying: Jailbreak Attack via Implicit Referencecited by 14
- 2025Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Studycited by 12
- 2025LLM-based HSE Compliance Assessment: Benchmark, Performance, and Advancementscited by 9
- 2025TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Lawcited by 8
- 2025Generative AI and LLMs in Industry: A text-mining Analysis and Critical Evaluation of Guidelines and Policy Statements Across Fourteen Industrial Sectorscited by 5
3 more
- 2025ELITE: Enhanced Language-Image Toxicity Evaluation for Safetycited by 5
- 2025Keep Security! Benchmarking Security Policy Preservation in Large Language Model Contexts Against Indirect Attacks in Question Answeringcited by 5
- 2025Unified Multi-Task Learning & Model Fusion for Efficient Language Model Guardrailingcited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Policy-Invisible Violations in LLM-Based Agents
- 2026ToolAlignBench: Investigating Alignment Conflicts in Tool-Calling Enabled LLMs
- 2026EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures
- 2026PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models
- 2025Safety Compliance: Rethinking LLM Safety Reasoning through the Lens of Compliance
- 2026What Matters For Safety Alignment?
- 2026PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?
- 2026aiXamine: Unified Black-Box Evaluation of Cross-Dimensional Trade-offs in LLM Safety, Security, and Privacy