# Evaluating Language Model Agency through Negotiations

arXiv: https://arxiv.org/abs/2401.04536 · HTML: https://mlsanity.com/paper/2401.04536/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 84 references that are in the map.
- First posted on arXiv 9 Jan 2024.

## Builds on (most cited first, 30 of 44)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [Adversarial Policies: Attacking Deep Reinforcement Learning](https://mlsanity.com/paper/1905.10615/) (2019)
- [Deal or No Deal? End-to-End Learning for Negotiation Dialogues](https://mlsanity.com/paper/1706.05125/) (2017)

## Built on by (most cited first, 30 of 48)

- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents](https://mlsanity.com/paper/2404.16698/) (2024)
- [Emergent social conventions and collective bias in LLM populations](https://mlsanity.com/paper/2410.08948/) (2024)
- [How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis](https://mlsanity.com/paper/2402.05863/) (2024)
- [Prediction-Powered Ranking of Large Language Models](https://mlsanity.com/paper/2402.17826/) (2024)
- [TextArena](https://mlsanity.com/paper/2504.11442/) (2025)
- [LLMs with Personalities in Multi-issue Negotiation Games](https://mlsanity.com/paper/2405.05248/) (2024)
- [STRIDE: A Tool-Assisted LLM Agent Framework for Strategic and Interactive Decision-Making](https://mlsanity.com/paper/2405.16376/) (2024)
- [Self-Recognition in Language Models](https://mlsanity.com/paper/2407.06946/) (2024)
- [Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontiers](https://mlsanity.com/paper/2502.09053/) (2025)
- [The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperation](https://mlsanity.com/paper/2511.02687/) (2025)
- [PLAYER*: Enhancing LLM-based Multi-Agent Communication and Interaction in Murder Mystery Games](https://mlsanity.com/paper/2404.17662/) (2024)
- [Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applications](https://mlsanity.com/paper/2505.00049/) (2025)
- [How Personality Traits Influence Negotiation Outcomes? A Simulation based on Large Language Models](https://mlsanity.com/paper/2407.11549/) (2024)
- [The Agentic Economy](https://mlsanity.com/paper/2505.15799/) (2025)
- [ARIA: Training Language Agents with Intention-Driven Reward Aggregation](https://mlsanity.com/paper/2506.00539/) (2025)
- [Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaboration](https://mlsanity.com/paper/2505.20625/) (2025)
- [Instructing LLMs to Negotiate using Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2604.09855/) (2026)
- [HARBOR: Exploring Persona Dynamics in Multi-Agent Competition](https://mlsanity.com/paper/2502.12149/) (2025)
- [Agent-to-Agent Theory of Mind: Testing Interlocutor Awareness among Large Language Models](https://mlsanity.com/paper/2506.22957/) (2025)
- [CORE: Measuring Multi-Agent LLM Interaction Quality under Game-Theoretic Pressures](https://mlsanity.com/paper/2508.11915/) (2025)
- [Reasoning-Driven Synthetic Data Generation and Evaluation](https://mlsanity.com/paper/2603.29791/) (2026)
- [Training Language Models for Bilateral Trade with Private Information](https://mlsanity.com/paper/2604.16472/) (2026)
- [Evaluating Multi-Turn Bargain Skills in LLM-Based Seller Agent](https://mlsanity.com/paper/2509.06341/) (2025)
- [Strategic Bargaining in Multi-Buyer Markets: Reinforcement Learning from Verifiable Rewards for LLM Negotiations](https://mlsanity.com/paper/2607.05863/) (2026)
- [ProMediate: A Socio-cognitive framework for evaluating proactive agents in multi-party negotiation](https://mlsanity.com/paper/2510.25224/) (2025)
- [Learning to Negotiate: Multi-Agent Deliberation for Collective Value Alignment in LLMs](https://mlsanity.com/paper/2603.10476/) (2026)
- [CRAFT: Grounded Multi-Agent Coordination Under Partial Information](https://mlsanity.com/paper/2603.25268/) (2026)
- [Interactive Evaluation Requires a Design Science](https://mlsanity.com/paper/2605.17829/) (2026)
- [Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMs](https://mlsanity.com/paper/2506.00577/) (2025)

## Abstract (arXiv)

We introduce an approach to evaluate language model (LM) agency using negotiation games. This approach better reflects real-world use cases and addresses some of the shortcomings of alternative LM benchmarks. Negotiation games enable us to study multi-turn, and cross-model interactions, modulate complexity, and side-step accidental evaluation data leakage. We use our approach to test six widely used and publicly accessible LMs, evaluating performance and alignment in both self-play and cross-play settings. Noteworthy findings include: (i) only closed-source models tested here were able to complete these tasks; (ii) cooperative bargaining games proved to be most challenging to the models; and (iii) even the most powerful models sometimes "lose" to weaker opponents