# COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMs

arXiv: https://arxiv.org/abs/2601.01836 · HTML: https://mlsanity.com/paper/2601.01836/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 89 references that are in the map.
- First posted on arXiv 5 Jan 2026.

## Builds on (most cited first, 15 of 15)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily](https://mlsanity.com/paper/2311.08268/) (2023)
- [A Complete Survey on LLM-based AI Chatbots](https://mlsanity.com/paper/2406.16937/) (2024)
- [Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements](https://mlsanity.com/paper/2410.08968/) (2024)
- [The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approaches](https://mlsanity.com/paper/2406.03339/) (2024)
- [Safety Through Reasoning: An Empirical Study of Reasoning Guardrail Models](https://mlsanity.com/paper/2505.20087/) (2025)
- [from Benign import Toxic: Jailbreaking the Language Model via Adversarial Metaphors](https://mlsanity.com/paper/2503.00038/) (2025)
- [You Know What I'm Saying: Jailbreak Attack via Implicit Reference](https://mlsanity.com/paper/2410.03857/) (2024)
- [Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Study](https://mlsanity.com/paper/2505.15389/) (2025)
- [LLM-based HSE Compliance Assessment: Benchmark, Performance, and Advancements](https://mlsanity.com/paper/2505.22959/) (2025)
- [TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law](https://mlsanity.com/paper/2507.21134/) (2025)
- [Generative AI and LLMs in Industry: A text-mining Analysis and Critical Evaluation of Guidelines and Policy Statements Across Fourteen Industrial Sectors](https://mlsanity.com/paper/2501.00957/) (2025)
- [ELITE: Enhanced Language-Image Toxicity Evaluation for Safety](https://mlsanity.com/paper/2502.04757/) (2025)
- [Keep Security! Benchmarking Security Policy Preservation in Large Language Model Contexts Against Indirect Attacks in Question Answering](https://mlsanity.com/paper/2505.15805/) (2025)
- [Unified Multi-Task Learning & Model Fusion for Efficient Language Model Guardrailing](https://mlsanity.com/paper/2504.19333/) (2025)

## Built on by (most cited first, 2 of 2)

- [When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Models](https://mlsanity.com/paper/2605.27851/) (2026)
- [Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge](https://mlsanity.com/paper/2610.07532/) (2026)

## Abstract (arXiv)

As large language models are deployed in high-stakes enterprise applications, from healthcare to finance, ensuring adherence to organization-specific policies has become essential. Yet existing safety evaluations focus exclusively on universal harms. We present COMPASS (Company/Organization Policy Alignment Assessment), the first systematic framework for evaluating whether LLMs comply with organizational allowlist and denylist policies. We apply COMPASS to eight diverse industry scenarios, generating and validating 5,920 queries that test both routine compliance and adversarial robustness through strategically designed edge cases. Evaluating seven state-of-the-art models, we uncover a fundamental asymmetry: models reliably handle legitimate requests (>95% accuracy) but catastrophically fail at enforcing prohibitions, refusing only 13-40% of adversarial denylist violations. These results demonstrate that current LLMs lack the robustness required for policy-critical deployments, establishing COMPASS as an essential evaluation framework for organizational AI safety.