PolicyKG: An Agentic LLM Pipeline for Translating Institutional Policies into SHACL Knowledge Graphs
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
Institutional policies stay in natural language while the systems that check compliance demand machine-readable constraints. Bridging that gap is still done by hand. PolicyKG closes the loop. It is an LLM pipeline that reads a policy PDF, classifies each sentence as an obligation, permission, or prohibition, lifts the label into first-order deontic logic, and emits SHACL constraints. Four stages run on a LangGraph state machine with per-stage validators. The piece that matters most is the Corpus Adapter: a YAML vocabulary registry that grounds LLM predicates in a target ontology. Retargeting to a new domain means swapping the registry, not retraining a model. On the Asian Institute of Technology Policies and Procedures corpus (1,663 sentences, 443 rules), PolicyKG reaches 86.9% deontic classification accuracy (Cohen's kappa = .709). Three annotators independently re-label a 50-item sample and agree at Fleiss' kappa = .844. SHACL shape correctness on a 69-shape subset is F1 = .866. The FOL path handles 79.2% of rules; the rest go through a direct NL-to-SHACL fallback. We audited every one of the 443 rules for second- or higher-order constructs. An automated regex checklist flagged none, and a first-author pass on the 92 FOL-fallback cases confirmed the same. The exact upper 95% Clopper-Pearson bound on the true HOL rate is 0.67%. This is an audit finding for one corpus, not a proof of FOL sufficiency for institutional policy. Swapping the AIT registry for a GDPR registry raises exact property alignment from 1/15 to 11/15 (Fisher's exact p < .001; Cohen's h = 1.53). On the LexDeMod lease-contract benchmark (N = 200), Macro F1 drops to .370 because lease English uses "shall be entitled" for permission -- exactly the vocabulary mismatch registry swap is meant to fix. Repeated runs produce hash-identical SHACL outputs.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 10 of its 22 references that are in the map. First posted on arXiv 10 Aug 2026.
Builds on
10 of 22 references.
References · 10
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2023Mistral 7Bcited by 3,065
- 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
- 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
- 2023Efficient Guided Generation for Large Language Modelscited by 192
- 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 186
- 2022Agent-Specific Deontic Modality Detection in Legal Languagecited by 7
- 2024Privacy Policy Analysis through Prompt Engineering for LLMscited by 6
- 2015The Rationale behind the Concept of Goalcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Knowledge Graph Representations for LLM-Based Policy Compliance Reasoning
- 2026PolicyBank: Evolving Policy Understanding for LLM Agents
- 2025GraphCompliance: Aligning Policy and Context Graphs for LLM-Based Regulatory Compliance
- 2025ARPaCCino: An Agentic-RAG for Policy as Code Compliance
- 2025Scaling Policy Compliance Assessment in Language Models with Policy Reasoning Traces
- 2026Policy-Invisible Violations in LLM-Based Agents
- 2026PolicyGuard: Prompt-Configurable Semantic DLP for LLM Coding Agents
- 2026AutoCedar: An Agentic Framework for Verifier-Guided Access Control Policy Synthesis