# Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agents

arXiv: https://arxiv.org/abs/2504.08640 · HTML: https://mlsanity.com/paper/2504.08640/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 59 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 20 of 20)

- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Large Language Models for Mathematical Reasoning: Progresses and Challenges](https://mlsanity.com/paper/2402.00157/) (2024)
- [Multi-Agent Risks from Advanced AI](https://mlsanity.com/paper/2502.14143/) (2025)
- [Frontier AI Regulation: Managing Emerging Risks to Public Safety](https://mlsanity.com/paper/2307.03718/) (2023)
- [Evaluating Language-Model Agents on Realistic Autonomous Tasks](https://mlsanity.com/paper/2312.11671/) (2023)
- [International AI Safety Report](https://mlsanity.com/paper/2501.17805/) (2025)
- [Evaluating the Consistency of LLM Evaluators](https://mlsanity.com/paper/2412.00543/) (2024)
- [The Role of Cooperation in Responsible AI Development](https://mlsanity.com/paper/1907.04534/) (2019)
- [The Brussels Effect and Artificial Intelligence: How EU regulation will impact the global AI market](https://mlsanity.com/paper/2208.12645/) (2022)
- [Zero-th Order Algorithm for Softmax Attention Optimization](https://mlsanity.com/paper/2307.08352/) (2023)
- [AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries](https://mlsanity.com/paper/2105.10356/) (2021)
- [When to (or not to) trust intelligent machines: Insights from an evolutionary game theory analysis of trust in repeated games](https://mlsanity.com/paper/2007.11338/) (2020)
- [Regulatory Markets for AI Safety](https://mlsanity.com/paper/2001.00078/) (2019)
- [Towards Mechanistic Interpretability of Graph Transformers via Attention Graphs](https://mlsanity.com/paper/2502.12352/) (2025)
- [Media and responsible AI governance: a game-theoretic and LLM analysis](https://mlsanity.com/paper/2503.09858/) (2025)
- [Trust AI Regulation? Discerning users are vital to build trust and effective AI regulation](https://mlsanity.com/paper/2403.09510/) (2024)
- [Playing games with Large language models: Randomness and strategy](https://mlsanity.com/paper/2503.02582/) (2025)
- [Large Language Models' Detection of Political Orientation in Newspapers](https://mlsanity.com/paper/2406.00018/) (2024)
- [Pathways to Good Healthcare Services and Patient Satisfaction: An Evolutionary Game Theoretical Approach](https://mlsanity.com/paper/1907.07132/) (2019)
- [RogueGPT: dis-ethical tuning transforms ChatGPT4 into a Rogue AI in 158 Words](https://mlsanity.com/paper/2407.15009/) (2024)

## Built on by (most cited first, 11 of 11)

- [Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets](https://mlsanity.com/paper/2510.25779/) (2025)
- [FAIRGAME: a Framework for AI Agents Bias Recognition using Game Theory](https://mlsanity.com/paper/2504.14325/) (2025)
- [Understanding LLM Agent Behaviours via Game Theory: Strategy Recognition, Biases and Multi-Agent Dynamics](https://mlsanity.com/paper/2512.07462/) (2025)
- [Strategic Communication and Language Bias in Multi-Agent LLM Coordination](https://mlsanity.com/paper/2508.00032/) (2025)
- [Can LLMs effectively provide game-theoretic-based scenarios for cybersecurity?](https://mlsanity.com/paper/2508.05670/) (2025)
- [Can Media Act as a Soft Regulator of Safe AI Development? A Game Theoretical Analysis](https://mlsanity.com/paper/2509.02650/) (2025)
- [Many-to-One Adversarial Consensus: Exposing Multi-Agent Collusion Risks in AI-Based Healthcare](https://mlsanity.com/paper/2512.03097/) (2025)
- [Payoff scaling shapes cooperation in LLM agents across languages](https://mlsanity.com/paper/2601.19082/) (2026)
- [Trust or Check? Understanding the (Evolutionary) Dynamics of User Trust in AI Systems](https://mlsanity.com/paper/2603.24742/) (2026)
- [A Multi-Agent Framework for Medical AI: Leveraging Fine-Tuned GPT, LLaMA, and DeepSeek R1 for Evidence-Based and Bias-Aware Clinical Query Processing](https://mlsanity.com/paper/2602.14158/) (2026)
- [The Two Genie Game: Adoption and Welfare in Audit-Grounded AI Governance](https://mlsanity.com/paper/2606.28710/) (2026)

## Abstract (arXiv)

There is general agreement that fostering trust and cooperation within the AI development ecosystem is essential to promote the adoption of trustworthy AI systems. By embedding Large Language Model (LLM) agents within an evolutionary game-theoretic framework, this paper investigates the complex interplay between AI developers, regulators and users, modelling their strategic choices under different regulatory scenarios. Evolutionary game theory (EGT) is used to quantitatively model the dilemmas faced by each actor, and LLMs provide additional degrees of complexity and nuances and enable repeated games and incorporation of personality traits. Our research identifies emerging behaviours of strategic AI agents, which tend to adopt more "pessimistic" (not trusting and defective) stances than pure game-theoretic agents. We observe that, in case of full trust by users, incentives are effective to promote effective regulation; however, conditional trust may deteriorate the "social pact". Establishing a virtuous feedback between users' trust and regulators' reputation thus appears to be key to nudge developers towards creating safe AI. However, the level at which this trust emerges may depend on the specific LLM used for testing. Our results thus provide guidance for AI regulation systems, and help predict the outcome of strategic LLM agents, should they be used to aid regulation itself.