# The Backfiring Effect of Weak AI Safety Regulation

arXiv: https://arxiv.org/abs/2503.20848 · HTML: https://mlsanity.com/paper/2503.20848/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 71 references that are in the map.
- First posted on arXiv 26 Mar 2025.

## Builds on (most cited first, 25 of 25)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Strategic Classification](https://mlsanity.com/paper/1506.06980/) (2015)
- [Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction](https://mlsanity.com/paper/2210.05791/) (2022)
- [Frontier AI Regulation: Managing Emerging Risks to Public Safety](https://mlsanity.com/paper/2307.03718/) (2023)
- [Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database](https://mlsanity.com/paper/2011.08512/) (2020)
- [Demystifying the Draft EU Artificial Intelligence Act](https://mlsanity.com/paper/2107.03721/) (2021)
- [International AI Safety Report](https://mlsanity.com/paper/2501.17805/) (2025)
- [Model-sharing Games: Analyzing Federated Learning Under Voluntary Participation](https://mlsanity.com/paper/2010.00753/) (2020)
- [On Evaluating the Durability of Safeguards for Open-Weight LLMs](https://mlsanity.com/paper/2412.07097/) (2024)
- [One for One, or All for All: Equilibria and Optimality of Collaboration in Federated Learning](https://mlsanity.com/paper/2103.03228/) (2021)
- [Measuring Representational Harms in Image Captioning](https://mlsanity.com/paper/2206.07173/) (2022)
- [Simple versus Optimal Contracts](https://mlsanity.com/paper/1808.03713/) (2018)
- [Stateful Strategic Regression](https://mlsanity.com/paper/2106.03827/) (2021)
- [Strategic Ranking](https://mlsanity.com/paper/2109.08240/) (2021)
- [A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms](https://mlsanity.com/paper/2407.01294/) (2024)
- [Fine-Tuning Games: Bargaining and Adaptation for General-Purpose Models](https://mlsanity.com/paper/2308.04399/) (2023)
- [Multi-Agent Contracts](https://mlsanity.com/paper/2211.05434/) (2022)
- [Accounting for AI and Users Shaping One Another: The Role of Mathematical Models](https://mlsanity.com/paper/2404.12366/) (2024)
- [Approximate Optimality of Linear Contracts Under Uncertainty](https://mlsanity.com/paper/2211.06850/) (2022)
- [In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI](https://mlsanity.com/paper/2503.16861/) (2025)
- [Selective Response Strategies for GenAI](https://mlsanity.com/paper/2502.00729/) (2025)
- [Multi-Agent Combinatorial Contracts](https://mlsanity.com/paper/2405.08260/) (2024)
- [Safety vs. Performance: How Multi-Objective Learning Reduces Barriers to Market Entry](https://mlsanity.com/paper/2409.03734/) (2024)

## Built on by (most cited first, 3 of 3)

- [Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Face](https://mlsanity.com/paper/2508.06811/) (2025)
- [Modeling the Economic Impacts of AI Openness Regulation](https://mlsanity.com/paper/2507.14193/) (2025)
- [Position: AI Governance Needs ISO-like Interoperability Protocols, Not Just Laws](https://mlsanity.com/paper/2608.14568/) (2026)

## Abstract (arXiv)

Recent policy proposals aim to improve the safety of general-purpose AI, but there is little understanding of the efficacy of different regulatory approaches to AI safety. We present a strategic model that explores the interactions between safety regulation, the general-purpose AI creators, and domain specialists--those who adapt the technology for specific applications. Our analysis examines how different regulatory measures, targeting different parts of the AI development chain, affect the outcome of this game. In particular, we assume AI technology is characterized by two key attributes: safety and performance. The regulator first sets a minimum safety standard that applies to one or both players, with strict penalties for non-compliance. The general-purpose creator then invests in the technology, establishing its initial safety and performance levels. Next, domain specialists refine the AI for their specific use cases, updating the safety and performance levels and taking the product to market. The resulting revenue is then distributed between the specialist and generalist through a revenue-sharing parameter. Our analysis reveals two key insights: First, weak safety regulation imposed predominantly on domain specialists can backfire. While it might seem logical to regulate AI use cases, our analysis shows that weak regulations targeting domain specialists alone can unintentionally reduce safety. This effect persists across a wide range of settings. Second, in sharp contrast to the previous finding, we observe that stronger, well-placed regulation can in fact mutually benefit all players subjected to it. When regulators impose appropriate safety standards on both general-purpose AI creators and domain specialists, the regulation functions as a commitment device, leading to safety and performance gains, surpassing what is achieved under no regulation or regulating one player alone.