Rules or Character? Scaling Laws for AI Safety Design
On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks
also in 1 more
Abstract (arXiv)
Artificial Intelligence (AI) safety systems combine character shaping (e.g., Reinforcement Learning from Human Feedback [RLHF], Constitutional AI), which modifies behavioral distributions at training time, with rule enforcement (e.g., output filters, safety classifiers), which blocks harmful outputs at inference time, yet little formal analysis exists on how their optimal balance should change as deployment scales increase. We introduce a stylized comparative-statics model that parameterizes safety design as a resource allocation alpha in [0,1] between these two approaches, incorporating scale-dependent filter degradation, common-mode failures, and character fragility -- the risk that shaped behavior degrades or collapses under novel conditions. Under a multiplicative Pareto damage model, we derive closed-form expected harm and supplement it with tail-risk (CVaR) analysis via Monte Carlo simulation. Across three scenarios (optimistic, moderate, pessimistic), the optimal alpha* is interior or at the rules-only boundary and shifts weakly toward character shaping as deployment scale T grows, from negligible (Delta alpha* = +0.01) to pronounced (Delta alpha* = +0.21) depending on scenario. The dominant parameter is the baseline character fragility rate p^(0)_frag, which shifts alpha* by 0.50 across its range -- far exceeding the effect of tail severity, filter quality, or common-mode failure probability. CVaR and expected-harm optima converge at large T. These results suggest that safety architecture decisions depend less on deployment scale per se than on the reliability of character shaping under distributional shift.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 5 of its 18 references that are in the map. First posted on arXiv 13 Aug 2026.
Builds on
5 of 18 references.
References · 5
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
- 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026When Safety Geometry Collapses: Fine-Tuning Vulnerabilities in Agentic Guard Models
- 2026Position: AI Safety Requires Effective Controllability
- 2026The Off-Support Barrier: Why Semantic Safety Constraints Are Not Learning-Problem Invariants, and What Follows for Prior Design, Containment, and Verification
- 2026Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety
- 2025From Refusal to Recovery: A Control-Theoretic Approach to Generative AI Guardrails
- 2025Noise Injection Systemically Degrades Large Language Model Safety Guardrails
- 2026AI Guardrail Survival under Single-Cycle Agentic Self-Summarization
- 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety