# Both eyes open: Vigilant Incentives help Regulatory Markets improve AI Safety

arXiv: https://arxiv.org/abs/2303.03174 · HTML: https://mlsanity.com/paper/2303.03174/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 76 references that are in the map.
- First posted on arXiv 6 Mar 2023.

## Builds on (most cited first, 25 of 25)

- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation](https://mlsanity.com/paper/1802.07228/) (2018)
- [Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims](https://mlsanity.com/paper/2004.07213/) (2020)
- [AI Safety Gridworlds](https://mlsanity.com/paper/1711.09883/) (2017)
- [The role of artificial intelligence in achieving the Sustainable Development Goals](https://mlsanity.com/paper/1905.00501/) (2019)
- [The structure and dynamics of multilayer networks](https://mlsanity.com/paper/1407.0742/) (2014)
- [The Role of Cooperation in Responsible AI Development](https://mlsanity.com/paper/1907.04534/) (2019)
- [The AI Index 2022 Annual Report](https://mlsanity.com/paper/2205.03468/) (2022)
- [The Brussels Effect and Artificial Intelligence: How EU regulation will impact the global AI market](https://mlsanity.com/paper/2208.12645/) (2022)
- [The Transformative Potential of Artificial Intelligence](https://mlsanity.com/paper/1912.00747/) (2019)
- [AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries](https://mlsanity.com/paper/2105.10356/) (2021)
- [Why and How Governments Should Monitor AI Development](https://mlsanity.com/paper/2108.12427/) (2021)
- [The Windfall Clause: Distributing the Benefits of AI for the Common Good](https://mlsanity.com/paper/1912.11595/) (2019)
- [Regulatory Markets for AI Safety](https://mlsanity.com/paper/2001.00078/) (2019)
- [System Cards for AI-Based Decision-Making for Public Policy](https://mlsanity.com/paper/2203.04754/) (2022)
- [The Offense-Defense Balance of Scientific Knowledge: Does Publishing AI Research Reduce Misuse?](https://mlsanity.com/paper/2001.00463/) (2019)
- [Voluntary safety commitments provide an escape from over-regulation in AI development](https://mlsanity.com/paper/2104.03741/) (2021)
- [Mediating Artificial Intelligence Developments through Negative and Positive Incentives](https://mlsanity.com/paper/2010.00403/) (2020)
- [Artificial Intelligence Development Races in Heterogeneous Settings](https://mlsanity.com/paper/2012.15234/) (2020)
- [The Tragedy of the AI Commons](https://mlsanity.com/paper/2006.05203/) (2020)
- [Actionable Guidance for High-Consequence AI Risk Management: Towards Standards Addressing AI Catastrophic Risks](https://mlsanity.com/paper/2206.08966/) (2022)
- [Social Diversity Reduces the Complexity and Cost of Fostering Fairness](https://mlsanity.com/paper/2211.10517/) (2022)
- [Institutional Incentives for the Evolution of Committed Cooperation: Ensuring Participation is as Important as Enhancing Compliance](https://mlsanity.com/paper/2110.13307/) (2021)
- [Targeting Interventions in Networks](https://mlsanity.com/paper/1710.06026/) (2017)
- [Should Artificial Intelligence Governance be Centralised? Design Lessons from History](https://mlsanity.com/paper/2001.03573/) (2020)

## Built on by (most cited first, 2 of 2)

- [Trust AI Regulation? Discerning users are vital to build trust and effective AI regulation](https://mlsanity.com/paper/2403.09510/) (2024)
- [Incentive Aware AI Regulations: A Credal Characterisation](https://mlsanity.com/paper/2603.05175/) (2026)

## Abstract (arXiv)

In the context of rapid discoveries by leaders in AI, governments must consider how to design regulation that matches the increasing pace of new AI capabilities. Regulatory Markets for AI is a proposal designed with adaptability in mind. It involves governments setting outcome-based targets for AI companies to achieve, which they can show by purchasing services from a market of private regulators. We use an evolutionary game theory model to explore the role governments can play in building a Regulatory Market for AI systems that deters reckless behaviour. We warn that it is alarmingly easy to stumble on incentives which would prevent Regulatory Markets from achieving this goal. These 'Bounty Incentives' only reward private regulators for catching unsafe behaviour. We argue that AI companies will likely learn to tailor their behaviour to how much effort regulators invest, discouraging regulators from innovating. Instead, we recommend that governments always reward regulators, except when they find that those regulators failed to detect unsafe behaviour that they should have. These 'Vigilant Incentives' could encourage private regulators to find innovative ways to evaluate cutting-edge AI systems.