# GLEE: A Unified Framework and Benchmark for Language-based Economic Environments

arXiv: https://arxiv.org/abs/2410.05254 · HTML: https://mlsanity.com/paper/2410.05254/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 54 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Builds on (most cited first, 24 of 24)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Playing repeated games with Large Language Models](https://mlsanity.com/paper/2305.16867/) (2023)
- [GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations](https://mlsanity.com/paper/2402.12348/) (2024)
- [How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis](https://mlsanity.com/paper/2402.05863/) (2024)
- [Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysis](https://mlsanity.com/paper/2312.05488/) (2023)
- [ChessGPT: Bridging Policy Learning and Language Modeling](https://mlsanity.com/paper/2306.09200/) (2023)
- [How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments](https://mlsanity.com/paper/2403.11807/) (2024)
- [CompeteAI: Understanding the Competition Dynamics in Large Language Model-based Agents](https://mlsanity.com/paper/2310.17512/) (2023)
- [Measuring Bargaining Abilities of LLMs: A Benchmark and A Buyer-Enhancement Method](https://mlsanity.com/paper/2402.15813/) (2024)
- [Modeling Content Creator Incentives on Algorithm-Curated Platforms](https://mlsanity.com/paper/2206.13102/) (2022)
- [Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk](https://mlsanity.com/paper/2401.05033/) (2024)
- [STEER: Assessing the Economic Rationality of Large Language Models](https://mlsanity.com/paper/2402.09552/) (2024)
- [LLMs with Personalities in Multi-issue Negotiation Games](https://mlsanity.com/paper/2405.05248/) (2024)
- [STRIDE: A Tool-Assisted LLM Agent Framework for Strategic and Interactive Decision-Making](https://mlsanity.com/paper/2405.16376/) (2024)
- [Economics Arena for Large Language Models](https://mlsanity.com/paper/2401.01735/) (2024)
- [Capturing the Complexity of Human Strategic Decision-Making with Machine Learning](https://mlsanity.com/paper/2408.07865/) (2024)
- [Can LLMs Replace Economic Choice Prediction Labs? The Case of Language-based Persuasion Games](https://mlsanity.com/paper/2401.17435/) (2024)
- [Human Choice Prediction in Language-based Persuasion Games: Simulation-based Off-Policy Evaluation](https://mlsanity.com/paper/2305.10361/) (2023)
- [Predicting Decisions in Language Based Persuasion Games](https://mlsanity.com/paper/2012.09966/) (2020)
- [Convergence of Learning Dynamics in Information Retrieval Games](https://mlsanity.com/paper/1806.05359/) (2018)
- [Designing an Automatic Agent for Repeated Language based Persuasion Games](https://mlsanity.com/paper/2105.04976/) (2021)
- [The Search for Stability: Learning Dynamics of Strategic Publishers with Initial Documents](https://mlsanity.com/paper/2305.16695/) (2023)
- [On the Convergence of No-Regret Dynamics in Information Retrieval Games with Proportional Ranking Functions](https://mlsanity.com/paper/2405.11517/) (2024)

## Built on by (most cited first, 20 of 20)

- [TMGBench: A Systematic Game Benchmark for Evaluating Strategic Reasoning Abilities of LLMs](https://mlsanity.com/paper/2410.10479/) (2024)
- [A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarios](https://mlsanity.com/paper/2412.03920/) (2024)
- [Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets](https://mlsanity.com/paper/2510.25779/) (2025)
- [Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontiers](https://mlsanity.com/paper/2502.09053/) (2025)
- [ARIA: Training Language Agents with Intention-Driven Reward Aggregation](https://mlsanity.com/paper/2506.00539/) (2025)
- [The Effect of State Representation on LLM Agent Behavior in Dynamic Routing Games](https://mlsanity.com/paper/2506.15624/) (2025)
- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [Alignment Makes Language Models Normative, Not Descriptive](https://mlsanity.com/paper/2603.17218/) (2026)
- [Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMs](https://mlsanity.com/paper/2506.00577/) (2025)
- [MALLES: A Multi-agent LLMs-based Economic Sandbox with Consumer Preference Alignment](https://mlsanity.com/paper/2603.17694/) (2026)
- [CSP: A Simulator For Multi-Agent Ranking Competitions](https://mlsanity.com/paper/2502.11197/) (2025)
- [Beyond Benchmarks: LLM Evaluation with an Anthropomorphic and Lifecycle-oriented Roadmap](https://mlsanity.com/paper/2508.18646/) (2025)
- [Sequential LLM Release Facilitates Manipulation in Regulated Markets](https://mlsanity.com/paper/2601.11496/) (2026)
- [BehaviorBench: Benchmarking Foundation Models for Behavioral Science Tasks](https://mlsanity.com/paper/2606.24162/) (2026)
- [RLRF: Competitive Search Agent Design via Reinforcement Learning from Ranker Feedback](https://mlsanity.com/paper/2510.04096/) (2025)
- [Predicting Decisions of AI Agents from Limited Interaction through Text-Tabular Modeling](https://mlsanity.com/paper/2605.12411/) (2026)
- [Can LLM Agents Price Competitively? A Dynamic Multi-Attribute Auction Benchmark for Agentic Commerce](https://mlsanity.com/paper/2608.00102/) (2026)
- [From Economic Agents to Agentic Economies: A Systems Blueprint for Economic World Models](https://mlsanity.com/paper/2608.06020/) (2026)
- [Benchmarking large language model agent societies against human behavioural distributions](https://mlsanity.com/paper/2608.28182/) (2026)
- [Receiver Decides Agent-to-Agent Negotiations](https://mlsanity.com/paper/2610.18424/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) show significant potential in economic and strategic interactions, where communication via natural language is often prevalent. This raises key questions: Do LLMs behave rationally? How do they perform compared to humans? Do they tend to reach an efficient and fair outcome? What is the role of natural language in strategic interaction? How do characteristics of the economic environment influence these dynamics? These questions become crucial concerning the economic and societal implications of integrating LLM-based agents into real-world data-driven systems, such as online retail platforms and recommender systems. To answer these questions, we introduce a benchmark for standardizing research on two-player, sequential, language-based games. Inspired by the economic literature, we define three base families of games with consistent parameterization, degrees of freedom and economic measures to evaluate agents' performance (self-gain), as well as the game outcome (efficiency and fairness). We develop an open-source framework for interaction simulation and analysis, and utilize it to collect a dataset of LLM vs. LLM interactions across numerous game configurations and an additional dataset of human vs. LLM interactions. Through extensive experimentation, we demonstrate how our framework and dataset can be used to: (i) compare the behavior of LLM-based agents in various economic contexts; (ii) evaluate agents in both individual and collective performance measures; and (iii) quantify the effect of the economic characteristics of the environments on the behavior of agents. Our results suggest that the market parameters, as well as the choice of the LLMs, tend to have complex and interdependent effects on the economic outcome, which calls for careful design and analysis of the language-based economic ecosystem.