# STEER: Assessing the Economic Rationality of Large Language Models

arXiv: https://arxiv.org/abs/2402.09552 · HTML: https://mlsanity.com/paper/2402.09552/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 24 of its 52 references that are in the map.
- First posted on arXiv 14 Feb 2024.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 24 of 24)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [Large Language Models as Tool Makers](https://mlsanity.com/paper/2305.17126/) (2023)
- [Tool Learning with Foundation Models](https://mlsanity.com/paper/2304.08354/) (2023)
- [Playing repeated games with Large Language Models](https://mlsanity.com/paper/2305.16867/) (2023)
- [WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain](https://mlsanity.com/paper/2211.00083/) (2022)
- [Towards Accurate Differential Diagnosis with Large Language Models](https://mlsanity.com/paper/2312.00164/) (2023)
- [Answering Questions by Meta-Reasoning over Multiple Chains of Thought](https://mlsanity.com/paper/2304.13007/) (2023)
- [Impact of News on the Commodity Market: Dataset and Results](https://mlsanity.com/paper/2009.04202/) (2020)

## Built on by (most cited first, 27 of 27)

- [Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs](https://mlsanity.com/paper/2502.08640/) (2025)
- [GLEE: A Unified Framework and Benchmark for Language-based Economic Environments](https://mlsanity.com/paper/2410.05254/) (2024)
- [Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets](https://mlsanity.com/paper/2510.25779/) (2025)
- [An Economy of AI Agents](https://mlsanity.com/paper/2509.01063/) (2025)
- [General agents contain world models](https://mlsanity.com/paper/2506.01622/) (2025)
- [Is Your LLM Overcharging You? Tokenization, Transparency, and Incentives](https://mlsanity.com/paper/2505.21627/) (2025)
- [AgenticPay: A Multi-Agent LLM Negotiation System for Buyer-Seller Transactions](https://mlsanity.com/paper/2602.06008/) (2026)
- [Strategic Tradeoffs Between Humans and AI in Multi-Agent Bargaining](https://mlsanity.com/paper/2509.09071/) (2025)
- [Evaluating the Ability of Large Language Models to Reason about Cardinal Directions](https://mlsanity.com/paper/2406.16528/) (2024)
- [EconEvals: Benchmarks and Litmus Tests for Economic Decision-Making by LLM Agents](https://mlsanity.com/paper/2503.18825/) (2025)
- [Do Large Language Models Know What They Are Capable Of?](https://mlsanity.com/paper/2512.24661/) (2025)
- [Are Large Language Models Sensitive to the Motives Behind Communication?](https://mlsanity.com/paper/2510.19687/) (2025)
- [AI Realtor: Towards Grounded Persuasive Language Generation for Automated Copywriting](https://mlsanity.com/paper/2502.16810/) (2025)
- [AI Agent Behavioral Science](https://mlsanity.com/paper/2506.06366/) (2025)
- [ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMs](https://mlsanity.com/paper/2510.00857/) (2025)
- [Investigating Tax Evasion Emergence Using Dual Large Language Model and Deep Reinforcement Learning Powered Agent-based Simulation](https://mlsanity.com/paper/2501.18177/) (2025)
- [Revealed Rationality: Label-Free Evaluation and Regularization from Representation Theorems](https://mlsanity.com/paper/2608.05015/) (2026)
- [Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMs](https://mlsanity.com/paper/2506.00577/) (2025)
- [Can LLMs Be CEOs? Benchmarking Strategic Resource Reallocation with Multi-Role Agent Simulation](https://mlsanity.com/paper/2606.17459/) (2026)
- [Cash or Comfort? How LLMs Value Your Inconvenience](https://mlsanity.com/paper/2506.17367/) (2025)
- [Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited](https://mlsanity.com/paper/2507.12059/) (2025)
- [How AI Systems Think About Education: Analyzing Latent Preference Patterns in Large Language Models](https://mlsanity.com/paper/2603.21006/) (2026)
- [Assessing mentalization in humans and large language models](https://mlsanity.com/paper/2608.26291/) (2026)
- [RLRF: Competitive Search Agent Design via Reinforcement Learning from Ranker Feedback](https://mlsanity.com/paper/2510.04096/) (2025)
- [PAVE: A Cognitive Architecture for Legitimate Violation in Generative Agent Societies](https://mlsanity.com/paper/2605.19351/) (2026)
- [Character Training for Risk-Averse Agents](https://mlsanity.com/paper/2609.38093/) (2026)
- [Choosing LLMs for Strategic Agent Interactions](https://mlsanity.com/paper/2610.18354/) (2026)

## Abstract (arXiv)

There is increasing interest in using LLMs as decision-making "agents." Doing so includes many degrees of freedom: which model should be used; how should it be prompted; should it be asked to introspect, conduct chain-of-thought reasoning, etc? Settling these questions -- and more broadly, determining whether an LLM agent is reliable enough to be trusted -- requires a methodology for assessing such an agent's economic rationality. In this paper, we provide one. We begin by surveying the economic literature on rational decision making, taxonomizing a large set of fine-grained "elements" that an agent should exhibit, along with dependencies between them. We then propose a benchmark distribution that quantitatively scores an LLMs performance on these elements and, combined with a user-provided rubric, produces a "STEER report card." Finally, we describe the results of a large-scale empirical experiment with 14 different LLMs, characterizing the both current state of the art and the impact of different model sizes on models' ability to exhibit rational behavior.