# Economics Arena for Large Language Models

arXiv: https://arxiv.org/abs/2401.01735 · HTML: https://mlsanity.com/paper/2401.01735/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jan 2024.

## Built on by (most cited first, 17 of 17)

- [From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents](https://mlsanity.com/paper/2412.03563/) (2024)
- [Can Large Language Model Agents Simulate Human Trust Behavior?](https://mlsanity.com/paper/2402.04559/) (2024)
- [A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarios](https://mlsanity.com/paper/2412.03920/) (2024)
- [GLEE: A Unified Framework and Benchmark for Language-based Economic Environments](https://mlsanity.com/paper/2410.05254/) (2024)
- [Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets](https://mlsanity.com/paper/2510.25779/) (2025)
- [Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontiers](https://mlsanity.com/paper/2502.09053/) (2025)
- [Can LLMs Replace Economic Choice Prediction Labs? The Case of Language-based Persuasion Games](https://mlsanity.com/paper/2401.17435/) (2024)
- [When Do Large Language Models Exhibit Unsolicited Deception?](https://mlsanity.com/paper/2504.00285/) (2025)
- [GATSim: Urban Mobility Simulation with Generative Agents](https://mlsanity.com/paper/2506.23306/) (2025)
- [EconWebArena: Benchmarking Autonomous Agents on Economic Tasks in Realistic Web Environments](https://mlsanity.com/paper/2506.08136/) (2025)
- [Approximating Human Strategic Reasoning with LLM-Enhanced Recursive Reasoners Leveraging Multi-agent Hypergames](https://mlsanity.com/paper/2502.07443/) (2025)
- [CHBench: A Cognitive Hierarchy Benchmark for Evaluating Strategic Reasoning Capability of LLMs](https://mlsanity.com/paper/2508.11944/) (2025)
- [BehaviorBench: Benchmarking Foundation Models for Behavioral Science Tasks](https://mlsanity.com/paper/2606.24162/) (2026)
- [Emergence of Reputation-Based Cooperation in LLM Agents](https://mlsanity.com/paper/2608.04507/) (2026)
- [RLRF: Competitive Search Agent Design via Reinforcement Learning from Ranker Feedback](https://mlsanity.com/paper/2510.04096/) (2025)
- [Not Yet: Humans Outperform LLMs in a Colonel Blotto Tournament](https://mlsanity.com/paper/2605.22095/) (2026)
- [Scaffolding the Strategist: Architecture-Dependent Reasoning Interventions in Hotelling Spatial Markets](https://mlsanity.com/paper/2607.09743/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have been extensively used as the backbones for general-purpose agents, and some economics literature suggest that LLMs are capable of playing various types of economics games. Following these works, to overcome the limitation of evaluating LLMs using static benchmarks, we propose to explore competitive games as an evaluation for LLMs to incorporate multi-players and dynamicise the environment. By varying the game history revealed to LLMs-based players, we find that most of LLMs are rational in that they play strategies that can increase their payoffs, but not as rational as indicated by Nash Equilibria (NEs). Moreover, when game history are available, certain types of LLMs, such as GPT-4, can converge faster to the NE strategies, which suggests higher rationality level in comparison to other models. In the meantime, certain types of LLMs can win more often when game history are available, and we argue that the winning rate reflects the reasoning ability with respect to the strategies of other players. Throughout all our experiments, we observe that the ability to strictly follow the game rules described by natural languages also vary among the LLMs we tested. In this work, we provide an economics arena for the LLMs research community as a dynamic simulation to test the above-mentioned abilities of LLMs, i.e. rationality, strategic reasoning ability, and instruction-following capability.