mlsanityData to 7 Oct 2026

Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback

arXiv 2305.10142 ↗First posted 17 May 2023cs.CLcited by 202

On the map: Synthetic data and self-improvement for LLMs → Self-play RL for reasoning

Abstract (arXiv)

We study whether multiple large language models (LLMs) can autonomously improve each other in a negotiation game by playing, reflecting, and criticizing. We are interested in this question because if LLMs were able to improve each other, it would imply the possibility of creating strong AI agents with minimal human intervention. We ask two LLMs to negotiate with each other, playing the roles of a buyer and a seller, respectively. They aim to reach a deal with the buyer targeting a lower price and the seller a higher one. A third language model, playing the critic, provides feedback to a player to improve the player's negotiation strategies. We let the two agents play multiple rounds, using previous negotiation history and AI feedback as in-context demonstrations to improve the model's negotiation strategy iteratively. We use different LLMs (GPT and Claude) for different roles and use the deal price as the evaluation metric. Our experiments reveal multiple intriguing findings: (1) Only a subset of the language models we consider can self-play and improve the deal price from AI feedback, weaker models either do not understand the game's rules or cannot incorporate AI feedback for further improvement. (2) Models' abilities to learn from the feedback differ when playing different roles. For example, it is harder for Claude-instant to improve as the buyer than as the seller. (3) When unrolling the game to multiple rounds, stronger agents can consistently improve their performance by meaningfully using previous experiences and iterative AI feedback, yet have a higher risk of breaking the deal. We hope our work provides insightful initial explorations of having models autonomously improve each other with game playing and AI feedback.

Key facts. Cited by 202 papers in the map (data through 7 Oct 2026). Placed in Self-play RL for reasoning, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 22 of its 34 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

22 of 34 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
10 more
  1. 2023Teaching Large Language Models to Self-Debugcited by 963
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  3. 2022Transformers learn in-context by gradient descentcited by 714
  4. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  5. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  6. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  7. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  8. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  9. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  10. 2021CaSiNo: A Corpus of Campsite Negotiation Dialogues for Automatic Negotiation Systemscited by 60

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 44; 2024: 79; 2025: 45; 2026: 34.

202 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  3. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  4. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 329
  5. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  6. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  7. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  8. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 246
  9. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  10. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  11. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 211
  12. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
38 more
  1. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  2. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 162
  3. 2023War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Warscited by 143
  4. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 129
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  6. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 122
  7. 2024Empowering Biomedical Discovery with AI Agentscited by 117
  8. 2023Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaborationcited by 115
  9. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 115
  10. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  11. 2023Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMscited by 103
  12. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 102
  13. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 97
  14. 2023Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiationcited by 94
  15. 2024AIOS: LLM Agent Operating Systemcited by 94
  16. 2024How Well Can LLMs Negotiate? NegotiationArena Platform and Analysiscited by 93
  17. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 92
  18. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 80
  19. 2024Towards Conversational Diagnostic AIcited by 79
  20. 2023Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arenacited by 78
  21. 2023Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communicationcited by 78
  22. 2023Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agentscited by 76
  23. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  24. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  25. 2024GroupDebate: Enhancing the Efficiency of Multi-Agent Debate Using Group Discussioncited by 64
  26. 2025Talk Isn't Always Cheap: Understanding Failure Modes in Multi-Agent Debatecited by 63
  27. 2023LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplaycited by 62
  28. 2023MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaborationcited by 56
  29. 2023SALMON: Self-Alignment with Instructable Reward Modelscited by 54
  30. 2024Learning to Use Tools via Cooperative and Interactive Agentscited by 54
  31. 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
  32. 2023Multi-Agent Consensus Seeking via Large Language Modelscited by 50
  33. 2023Let Models Speak Ciphers: Multiagent Debate through Embeddingscited by 49
  34. 2023Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Modelscited by 48
  35. 2024Towards Human-centered Proactive Conversational Agentscited by 48
  36. 2024Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Modelscited by 47
  37. 2023Large Language Models are Zero Shot Hypothesis Proposerscited by 45
  38. 2024GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Modelscited by 44

50 most cited of 202 All 202 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024AgreeMate: Teaching LLMs to Haggle
  2. 2026Do Small Language Models Learn to Negotiate? A Controlled Scaling Study of RL-Trained Sellers
  3. 2025The Price of Thought: A Multilingual Analysis of Reasoning, Performance, and Cost of Negotiation in Large Language Models
  4. 2026Training Language Models for Bilateral Trade with Private Information
  5. 2025Strategic Tradeoffs Between Humans and AI in Multi-Agent Bargaining
  6. 2025The Illusion of Rationality: Tacit Bias and Strategic Dominance in Frontier LLM Negotiation Games
  7. 2026Learning to Sell: Reinforcement Learning for Strategic Large Language Model Agents in Multi-Product Markets
  8. 2023Strategic Reasoning with Language Models