mlsanityData to 7 Oct 2026

STEER: Assessing the Economic Rationality of Large Language Models

arXiv 2402.09552 ↗First posted 14 Feb 2024cs.CL econ.GN q-fin.ECcited by 26

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

There is increasing interest in using LLMs as decision-making "agents." Doing so includes many degrees of freedom: which model should be used; how should it be prompted; should it be asked to introspect, conduct chain-of-thought reasoning, etc? Settling these questions -- and more broadly, determining whether an LLM agent is reliable enough to be trusted -- requires a methodology for assessing such an agent's economic rationality. In this paper, we provide one. We begin by surveying the economic literature on rational decision making, taxonomizing a large set of fine-grained "elements" that an agent should exhibit, along with dependencies between them. We then propose a benchmark distribution that quantitatively scores an LLMs performance on these elements and, combined with a user-provided rubric, produces a "STEER report card." Finally, we describe the results of a large-scale empirical experiment with 14 different LLMs, characterizing the both current state of the art and the impact of different model sizes on models' ability to exhibit rational behavior.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 24 of its 52 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

24 of 52 references.

References · 24

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
12 more
  1. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  2. 2022Large Language Models Can Self-Improvecited by 656
  3. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  4. 2023The Falcon Series of Open Language Modelscited by 526
  5. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 378
  6. 2023Tool Learning with Foundation Modelscited by 302
  7. 2023Large Language Models as Tool Makerscited by 293
  8. 2023Playing repeated games with Large Language Modelscited by 213
  9. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
  10. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 115
  11. 2023Answering Questions by Meta-Reasoning over Multiple Chains of Thoughtcited by 87
  12. 2020Impact of News on the Commodity Market: Dataset and Resultscited by 80

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 17; 2026: 7.

26 papers.

Sort:

By area:

  1. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 63
  2. 2024GLEE: A Unified Framework and Benchmark for Language-based Economic Environmentscited by 19
  3. 2025An Economy of AI Agentscited by 17
  4. 2025Magentic Marketplace: An Open-Source Environment for Studying Agentic Marketscited by 15
  5. 2025General agents contain world modelscited by 14
  6. 2025Is Your LLM Overcharging You? Tokenization, Transparency, and Incentivescited by 13
  7. 2025Strategic Tradeoffs Between Humans and AI in Multi-Agent Bargainingcited by 11
  8. 2026AgenticPay: A Multi-Agent LLM Negotiation System for Buyer-Seller Transactionscited by 11
  9. 2024Evaluating the Ability of Large Language Models to Reason about Cardinal Directionscited by 10
  10. 2025Do Large Language Models Know What They Are Capable Of?cited by 9
  11. 2025EconEvals: Benchmarks and Litmus Tests for Economic Decision-Making by LLM Agentscited by 8
  12. 2025Are Large Language Models Sensitive to the Motives Behind Communication?cited by 7
14 more
  1. 2025AI Realtor: Towards Grounded Persuasive Language Generation for Automated Copywritingcited by 6
  2. 2025AI Agent Behavioral Sciencecited by 5
  3. 2025Investigating Tax Evasion Emergence Using Dual Large Language Model and Deep Reinforcement Learning Powered Agent-based Simulationcited by 3
  4. 2025ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMscited by 3
  5. 2026Revealed Rationality: Label-Free Evaluation and Regularization from Representation Theoremscited by 3
  6. 2025Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMscited by 2
  7. 2026Can LLMs Be CEOs? Benchmarking Strategic Resource Reallocation with Multi-Role Agent Simulationcited by 2
  8. 2025Cash or Comfort? How LLMs Value Your Inconveniencecited by 1
  9. 2025Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisitedcited by 1
  10. 2026How AI Systems Think About Education: Analyzing Latent Preference Patterns in Large Language Modelscited by 1
  11. 2025RLRF: Competitive Search Agent Design via Reinforcement Learning from Ranker Feedbackcited by 0
  12. 2026PAVE: A Cognitive Architecture for Legitimate Violation in Generative Agent Societiescited by 0
  13. 2026Assessing mentalization in humans and large language modelscited by 0
  14. 2026Character Training for Risk-Averse Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025STEER-ME: Assessing the Microeconomic Reasoning of Large Language Models
  2. 2025Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Making
  3. 2026LLM Consumer Behavior Theory: Foundations of a Novel Research Field
  4. 2024Comparing Rationality Between Large Language Models and Humans: Insights and Open Questions
  5. 2026Competitive Market Behavior of LLMs
  6. 2025Rationales Are Not Silver Bullets: Measuring the Impact of Rationales on Model Performance and Reliability
  7. 2026Mind the (DH) Gap! A Contrast in Risky Choices Between Reasoning and Conversational LLMs
  8. 2024Towards Rationality in Language and Multimodal Agents: A Survey