# Trust or Check? Understanding the (Evolutionary) Dynamics of User Trust in AI Systems

arXiv: https://arxiv.org/abs/2603.24742 · HTML: https://mlsanity.com/paper/2603.24742/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 56 references that are in the map.
- First posted on arXiv 25 Mar 2026.

## Builds on (most cited first, 18 of 18)

- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [Multi-Agent Risks from Advanced AI](https://mlsanity.com/paper/2502.14143/) (2025)
- [A multi-agent reinforcement learning model of common-pool resource appropriation](https://mlsanity.com/paper/1707.06600/) (2017)
- [The impact of generative artificial intelligence on socioeconomic inequalities and policy making](https://mlsanity.com/paper/2401.05377/) (2023)
- [When to (or not to) trust intelligent machines: Insights from an evolutionary game theory analysis of trust in repeated games](https://mlsanity.com/paper/2007.11338/) (2020)
- [Media and responsible AI governance: a game-theoretic and LLM analysis](https://mlsanity.com/paper/2503.09858/) (2025)
- [Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agents](https://mlsanity.com/paper/2504.08640/) (2025)
- [Voluntary safety commitments provide an escape from over-regulation in AI development](https://mlsanity.com/paper/2104.03741/) (2021)
- [Trust AI Regulation? Discerning users are vital to build trust and effective AI regulation](https://mlsanity.com/paper/2403.09510/) (2024)
- [Investigating the Impact of Direct Punishment on the Emergence of Cooperation in Multi-Agent Reinforcement Learning Systems](https://mlsanity.com/paper/2301.08278/) (2023)
- [Social physics in the age of artificial intelligence](https://mlsanity.com/paper/2603.16900/) (2026)
- [Pathways to Good Healthcare Services and Patient Satisfaction: An Evolutionary Game Theoretical Approach](https://mlsanity.com/paper/1907.07132/) (2019)
- [Decoding trust: A reinforcement learning perspective](https://mlsanity.com/paper/2309.14598/) (2023)
- [Learning Fair Cooperation in Mixed-Motive Games with Indirect Reciprocity](https://mlsanity.com/paper/2408.04549/) (2024)
- [Can Media Act as a Soft Regulator of Safe AI Development? A Game Theoretical Analysis](https://mlsanity.com/paper/2509.02650/) (2025)
- [Co-evolutionary Dynamics of Attack and Defence in Cybersecurity](https://mlsanity.com/paper/2505.19338/) (2025)
- [Disentangling trust from cooperation: Evolution of trust as reduced monitoring in social dilemmas](https://mlsanity.com/paper/2509.04143/) (2025)
- [Cooperation Through Indirect Reciprocity in Child-Robot Interactions](https://mlsanity.com/paper/2512.20621/) (2025)

## Built on by (most cited first, 1 of 1)

- [Strategic commitments shape collective cybersecurity under AI inequality](https://mlsanity.com/paper/2605.09415/) (2026)

## Abstract (arXiv)

As the capabilities and adoption of Artificial Intelligence (AI) systems grow, trust in these AI systems is an increasingly urgent concern. Much research has focused on models of AI governance and has primarily examined incentives for safe development and effective regulation. Hence they typically represented users trust as a one-shot adoption choice rather than as a dynamic, evolving process shaped by repeated interactions. We instead model trust as the dynamic choice of reduced monitoring in a repeated, asymmetric interaction between users and AI developers, where checking developers' behaviour is costly. Using evolutionary game theory, we study how users' strategies of trust and developers' strategies of providing safe (compliant) or unsafe (non-compliant) AI co-evolve under different levels of monitoring cost and institutional regimes. We conduct the analysis on both imitation-based and learning-based perspectives, with the stochastic finite-population dynamics, the infinite-population replicator analysis and the reinforcement learning analysis. We find three robust long-run regimes: no adoption by users while developers provide unsafe AI, unsafe but widely adopted systems, and safe systems that are widely adopted. Only the last is desirable, and it arises when penalties for unsafe behaviour exceed the extra cost of safety and users can still afford to monitor at least occasionally. Our results formally support governance proposals that emphasise transparency, low-cost monitoring, and meaningful sanctions, and they show that neither regulation alone nor blind user trust is sufficient to prevent the drift towards unsafe or low-adoption outcomes.