mlsanityData to 14 Oct 2026

Why Do AI Agents Break Rules? How Framing, Context, and Social Signals Shape Compliance

arXiv 2608.12323 ↗First posted 29 May 2026cs.CL cs.AI cs.CYcited by 4

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Specifying a penalty can turn a legal obligation into a cost-benefit calculation that favors violation. We show that this enforcement information paradox occurs in AI agents. Most AI safety evaluations test whether models fail; we ask why, using compliance theory from law and economics as a diagnostic. We evaluate twelve instruction-tuned language models deployed as enterprise procurement chatbots. Each is given an environmental regulation in its system prompt covering large purchases, and a vendor list on which the certified suppliers cost nearly twice what the uncertified ones do. We test the agents against the predictions of deterrence, legitimacy, and expressive law, and find that each theory accounts for part of what we observe. Under identical conditions, compliance spans 46 percentage points across models, and models differ in which pressure breaks them: some treat the regulation as binding however it is worded, while others fail where theory predicts, under low penalties and non-command phrasing. Benchmark scores and developers' own descriptions of post-training do not predict where a model falls. Across all twelve, financial incentives, managerial demands, peer outcomes, and employee pressure each produce large compliance failures. These agents violate regulatory constraints to satisfy local user objectives in ways standard alignment benchmarks do not measure. Embedding the rule in the system prompt is not on its own enough to produce a compliant agent: model selection is itself a governance decision, and benchmark evaluation is not sufficient for compliance-sensitive deployments.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 8 of its 29 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

8 of 29 references.

References · 8

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  5. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  6. 2024Frontier Models are Capable of In-context Schemingcited by 287
  7. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 146
  8. 2026AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviorscited by 38

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026AI Alignment via Incentives and Correction
  2. 2023Of Models and Tin Men: A Behavioural Economics Study of Principal-Agent Problems in AI Alignment using Large-Language Models
  3. 2026Norm Enforcement for AI Agents: Robustly Shaping Behavior in Multi-Agent Systems
  4. 2026Agentic AI, Retrieval-Augmented Generation, and the Institutional Turn: Legal Architectures and Financial Governance in the Age of Distributional AGI
  5. 2026A pragmatic approach to regulating AI agents
  6. 2026Why Agents Compromise Safety Under Pressure
  7. 2026Policy-Invisible Violations in LLM-Based Agents
  8. 2025AI Agents and the Law