# Of Models and Tin Men: A Behavioural Economics Study of Principal-Agent Problems in AI Alignment using Large-Language Models arXiv: https://arxiv.org/abs/2307.11137 ยท HTML: https://mlsanity.com/paper/2307.11137/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. - Builds on 19 of its 39 references that are in the map. - First posted on arXiv 20 Jul 2023. ## Placed in - LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/) ## Builds on (most cited first, 19 of 19) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017) - [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023) - [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023) - [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022) - [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016) - [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020) - [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018) - [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022) - [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018) - [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019) - [Playing repeated games with Large Language Models](https://mlsanity.com/paper/2305.16867/) (2023) - [Evidence of behavior consistent with self-interest and altruism in an artificially intelligent agent](https://mlsanity.com/paper/2301.02330/) (2023) - [Measuring an artificial intelligence agent's trust in humans using machine incentives](https://mlsanity.com/paper/2212.13371/) (2022) - [Population-based Evaluation in Repeated Rock-Paper-Scissors as a Benchmark for Multiagent Reinforcement Learning](https://mlsanity.com/paper/2303.03196/) (2023) ## Built on by (most cited first, 9 of 9) - [Position: Towards a Responsible LLM-empowered Multi-Agent Systems](https://mlsanity.com/paper/2502.01714/) (2025) - [Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Models](https://mlsanity.com/paper/2507.07484/) (2025) - [Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignment](https://mlsanity.com/paper/2510.14008/) (2025) - [LLM agents security duality: a comprehensive survey of self-security and empowered cybersecurity](https://mlsanity.com/paper/2606.28450/) (2026) - [Multi-Agent Systems Should be Treated as Principal-Agent Problems](https://mlsanity.com/paper/2601.23211/) (2026) - [Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective](https://mlsanity.com/paper/2602.02572/) (2026) - [On-Policy Distillation for LLM Safety: A Routing Approach to Template-Robust Realignment](https://mlsanity.com/paper/2607.27081/) (2026) - [Behavioural Analysis of Alignment Faking](https://mlsanity.com/paper/2605.27681/) (2026) - [How to Detect and Measure the AI Dangers to Democracy](https://mlsanity.com/paper/2606.16054/) (2026) ## Abstract (arXiv) AI Alignment is often presented as an interaction between a single designer and an artificial agent in which the designer attempts to ensure the agent's behavior is consistent with its purpose, and risks arise solely because of conflicts caused by inadvertent misalignment between the utility function intended by the designer and the resulting internal utility function of the agent. With the advent of agents instantiated with large-language models (LLMs), which are typically pre-trained, we argue this does not capture the essential aspects of AI safety because in the real world there is not a one-to-one correspondence between designer and agent, and the many agents, both artificial and human, have heterogeneous values. Therefore, there is an economic aspect to AI safety and the principal-agent problem is likely to arise. In a principal-agent problem conflict arises because of information asymmetry together with inherent misalignment between the utility of the agent and its principal, and this inherent misalignment cannot be overcome by coercing the agent into adopting a desired utility function through training. We argue the assumptions underlying principal-agent problems are crucial to capturing the essence of safety problems involving pre-trained AI models in real-world situations. Taking an empirical approach to AI safety, we investigate how GPT models respond in principal-agent conflicts. We find that agents based on both GPT-3.5 and GPT-4 override their principal's objectives in a simple online shopping task, showing clear evidence of principal-agent conflict. Surprisingly, the earlier GPT-3.5 model exhibits more nuanced behaviour in response to changes in information asymmetry, whereas the later GPT-4 model is more rigid in adhering to its prior alignment. Our results highlight the importance of incorporating principles from economics into the alignment process.