mlsanityData to 7 Oct 2026

Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena

arXiv 2310.05746 ↗First posted 9 Oct 2023cs.CL cs.AIcited by 78

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) showcase advanced reasoning, yet NLP evaluations often depend on static benchmarks. Evaluating this necessitates environments that test strategic reasoning in dynamic, competitive scenarios requiring long-term planning. We introduce AucArena, a novel evaluation suite that simulates auctions, a setting chosen for being highly unpredictable and involving many skills related to resource and risk management, while also being easy to evaluate. We conduct controlled experiments using state-of-the-art LLMs to power bidding agents to benchmark their planning and execution skills. Our research demonstrates that LLMs, such as GPT-4, possess key skills for auction participation, such as budget management and goal adherence, which improve with adaptive strategies. This highlights LLMs' potential in modeling complex social interactions in competitive contexts. However, variability in LLM performance and occasional outperformance by simpler methods indicate opportunities for further advancements in LLM design and the value of our simulation environment for ongoing testing and refinement.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 43 of its 95 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

43 of 95 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  12. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
31 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  10. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  11. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  13. 2023PaLM 2 Technical Reportcited by 1,154
  14. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  15. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  16. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  17. 2023Cognitive Architectures for Language Agentscited by 422
  18. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  19. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  20. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  21. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  22. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 161
  23. 2022Language Models as Agent Modelscited by 131
  24. 2023Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Trackercited by 95
  25. 2023Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agentscited by 52
  26. 2023Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Modelscited by 46
  27. 2023Distilling Script Knowledge from Large Language Models for Constrained Language Planningcited by 45
  28. 2022Towards Teachable Reasoning Systems: Using a Dynamic Memory of User Feedback for Continual System Improvementcited by 37
  29. 2023Interactive Natural Language Processingcited by 31
  30. 2022Breakpoint Transformers for Modeling and Tracking Intermediate Beliefscited by 6
  31. 2021Natural-Language Multi-Agent Simulations of Argumentative Opinion Dynamicscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 29; 2025: 31; 2026: 12.

78 papers.

Sort:

By area:

  1. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  2. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  3. 2023EconAgent: Large Language Model-Empowered Agents for Simulating Macroeconomic Activitiescited by 154
  4. 2023War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Warscited by 143
  5. 2024EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithmscited by 109
  6. 2025Agentic Large Language Models, a surveycited by 105
  7. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 97
  8. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agentscited by 89
  9. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 86
  10. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
  11. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  12. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environmentscited by 66
38 more
  1. 2023CompeteAI: Understanding the Competition Dynamics in Large Language Model-based Agentscited by 65
  2. 2024A Survey on LLM-based Multi-Agent System: Recent Advances and New Frontiers in Applicationcited by 54
  3. 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 45
  4. 2024Measuring Bargaining Abilities of LLMs: A Benchmark and A Buyer-Enhancement Methodcited by 44
  5. 2024Cocoa: Co-Planning and Co-Execution with AI Agentscited by 38
  6. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  7. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environmentscited by 35
  8. 2024Agent Design Pattern Catalogue: A Collection of Architectural Patterns for Foundation Model based Agentscited by 35
  9. 2023ALYMPICS: LLM Agents Meet Game Theory -- Exploring Strategic Decision-Making with AI Agentscited by 33
  10. 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agentscited by 32
  11. 2024When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environmentscited by 25
  12. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24
  13. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  14. 2024TimeArena: Shaping Efficient Multitasking Language Agents in a Time-Aware Simulationcited by 22
  15. 2024SelfGoal: Your Language Agents Already Know How to Achieve High-level Goalscited by 21
  16. 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agentscited by 16
  17. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
  18. 2025Magentic Marketplace: An Open-Source Environment for Studying Agentic Marketscited by 15
  19. 2024RENOVI: A Benchmark Towards Remediating Norm Violations in Socio-Cultural Conversationscited by 14
  20. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 14
  21. 2023Towards Responsible Generative AI: A Reference Architecture for Designing Foundation Model based Agentscited by 13
  22. 2025LiveTradeBench: Seeking Real-World Alpha with Large Language Modelscited by 13
  23. 2024What if LLMs Have Different World Views: Simulating Alien Civilizations with LLM-based Agentscited by 12
  24. 2024Explaining Large Language Models Decisions Using Shapley Valuescited by 12
  25. 2024Helmsman of the Masses? Evaluate the Opinion Leadership of Large Language Models in the Werewolf Gamecited by 12
  26. 2024SurveyAgent: A Conversational System for Personalized and Efficient Research Surveycited by 11
  27. 2026AgenticPay: A Multi-Agent LLM Negotiation System for Buyer-Seller Transactionscited by 11
  28. 2025Mimicking the Familiar: Dynamic Command Generation for Information Theft Attacks in LLM Tool-Learning Systemcited by 10
  29. 2025Can Generative AI agents behave like humans? Evidence from laboratory market experimentscited by 10
  30. 2025LLM Bidders Preserve the Mechanism-Level Orderings of Human Bidderscited by 10
  31. 2024LASP: Surveying the State-of-the-Art in Large Language Model-Assisted AI Planningcited by 9
  32. 2025PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attentioncited by 8
  33. 2025EconEvals: Benchmarks and Litmus Tests for Economic Decision-Making by LLM Agentscited by 8
  34. 2024Look Further Ahead: Testing the Limits of GPT-4 in Path Planningcited by 7
  35. 2025Towards Strategic Persuasion with Language Modelscited by 7
  36. 2024Will GPT-4 Run DOOM?cited by 6
  37. 2024AgentGroupChat: An Interactive Group Chat Simulacra For Better Eliciting Emergent Behaviorcited by 6
  38. 2025HARBOR: Exploring Persona Dynamics in Multi-Agent Competitioncited by 6

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024STRIDE: A Tool-Assisted LLM Agent Framework for Strategic and Interactive Decision-Making
  2. 2026Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining
  3. 2025Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making
  4. 2026Agora: Enhancing LLM Agent Reasoning Via Auction-Based Task Allocation
  5. 2026GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models
  6. 2024Reproducing and Extending Experiments in Behavioral Strategy with Large Language Models
  7. 2026Agentic Trading: When LLM Agents Meet Financial Markets
  8. 2026Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments