mlsanityData to 7 Oct 2026

Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agents

arXiv 2504.08640 ↗First posted 11 Apr 2025cs.AI cs.CY cs.GTcited by 11

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

There is general agreement that fostering trust and cooperation within the AI development ecosystem is essential to promote the adoption of trustworthy AI systems. By embedding Large Language Model (LLM) agents within an evolutionary game-theoretic framework, this paper investigates the complex interplay between AI developers, regulators and users, modelling their strategic choices under different regulatory scenarios. Evolutionary game theory (EGT) is used to quantitatively model the dilemmas faced by each actor, and LLMs provide additional degrees of complexity and nuances and enable repeated games and incorporation of personality traits. Our research identifies emerging behaviours of strategic AI agents, which tend to adopt more "pessimistic" (not trusting and defective) stances than pure game-theoretic agents. We observe that, in case of full trust by users, incentives are effective to promote effective regulation; however, conditional trust may deteriorate the "social pact". Establishing a virtuous feedback between users' trust and regulators' reputation thus appears to be key to nudge developers towards creating safe AI. However, the level at which this trust emerges may depend on the specific LLM used for testing. Our results thus provide guidance for AI regulation systems, and help predict the outcome of strategic LLM agents, should they be used to aid regulation itself.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 20 of its 59 references that are in the map. First posted on arXiv 11 Apr 2025.

Builds on

20 of 59 references.

References · 20

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  3. 2025Multi-Agent Risks from Advanced AIcited by 150
  4. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  5. 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 94
  6. 2025International AI Safety Reportcited by 64
  7. 2024Evaluating the Consistency of LLM Evaluatorscited by 24
  8. 2019The Role of Cooperation in Responsible AI Developmentcited by 21
  9. 2022The Brussels Effect and Artificial Intelligence: How EU regulation will impact the global AI marketcited by 18
  10. 2023Zero-th Order Algorithm for Softmax Attention Optimizationcited by 16
  11. 2021AI Certification: Advancing Ethical Practice by Reducing Information Asymmetriescited by 15
  12. 2020When to (or not to) trust intelligent machines: Insights from an evolutionary game theory analysis of trust in repeated gamescited by 14
8 more
  1. 2019Regulatory Markets for AI Safetycited by 13
  2. 2025Towards Mechanistic Interpretability of Graph Transformers via Attention Graphscited by 12
  3. 2025Media and responsible AI governance: a game-theoretic and LLM analysiscited by 11
  4. 2024Trust AI Regulation? Discerning users are vital to build trust and effective AI regulationcited by 9
  5. 2025Playing games with Large language models: Randomness and strategycited by 7
  6. 2024Large Language Models' Detection of Political Orientation in Newspaperscited by 5
  7. 2019Pathways to Good Healthcare Services and Patient Satisfaction: An Evolutionary Game Theoretical Approachcited by 4
  8. 2024RogueGPT: dis-ethical tuning transforms ChatGPT4 into a Rogue AI in 158 Wordscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 4.

11 papers.

  1. 2025Magentic Marketplace: An Open-Source Environment for Studying Agentic Marketscited by 15
  2. 2025FAIRGAME: a Framework for AI Agents Bias Recognition using Game Theorycited by 13
  3. 2025Understanding LLM Agent Behaviours via Game Theory: Strategy Recognition, Biases and Multi-Agent Dynamicscited by 11
  4. 2025Strategic Communication and Language Bias in Multi-Agent LLM Coordinationcited by 3
  5. 2025Can Media Act as a Soft Regulator of Safe AI Development? A Game Theoretical Analysiscited by 3
  6. 2025Many-to-One Adversarial Consensus: Exposing Multi-Agent Collusion Risks in AI-Based Healthcarecited by 3
  7. 2025Can LLMs effectively provide game-theoretic-based scenarios for cybersecurity?cited by 2
  8. 2026Payoff scaling shapes cooperation in LLM agents across languagescited by 1
  9. 2026Trust or Check? Understanding the (Evolutionary) Dynamics of User Trust in AI Systemscited by 1
  10. 2026A Multi-Agent Framework for Medical AI: Leveraging Fine-Tuned GPT, LLaMA, and DeepSeek R1 for Evidence-Based and Bias-Aware Clinical Query Processingcited by 0
  11. 2026The Two Genie Game: Adoption and Welfare in Audit-Grounded AI Governancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025When Trust Collides: Decoding Human-LLM Cooperation Dynamics through the Prisoner's Dilemma
  2. 2024Can Large Language Model Agents Simulate Human Trust Behavior?
  3. 2025A closer look at how large language models trust humans: patterns and biases
  4. 2022Measuring an artificial intelligence agent's trust in humans using machine incentives
  5. 2024Can We Trust AI Agents? A Case Study of an LLM-Based Multi-Agent System for Ethical AI
  6. 2026Evaluating Collective Behaviour of Hundreds of LLM Agents
  7. 2025Evaluating LLMs in Open-Source Games
  8. 2025Regulating the Agency of LLM-based Agents