# NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding

arXiv: https://arxiv.org/abs/2404.13627 · HTML: https://mlsanity.com/paper/2404.13627/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 104 references that are in the map.
- First posted on arXiv 21 Apr 2024.

## Builds on (most cited first, 30 of 38)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://mlsanity.com/paper/2210.13312/) (2022)
- [Deal or No Deal? End-to-End Learning for Negotiation Dialogues](https://mlsanity.com/paper/1706.05125/) (2017)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023)
- [Leveraging Large Language Models for Multiple Choice Question Answering](https://mlsanity.com/paper/2210.12353/) (2022)
- [Time Travel in LLMs: Tracing Data Contamination in Large Language Models](https://mlsanity.com/paper/2308.08493/) (2023)
- [Understanding Social Reasoning in Language Models with Language Models](https://mlsanity.com/paper/2306.15448/) (2023)
- [FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions](https://mlsanity.com/paper/2310.15421/) (2023)
- [Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models](https://mlsanity.com/paper/2305.14763/) (2023)
- [Decoupling Strategy and Generation in Negotiation Dialogues](https://mlsanity.com/paper/1808.09637/) (2018)
- [Task Contamination: Language Models May Not Be Few-Shot Anymore](https://mlsanity.com/paper/2312.16337/) (2023)
- [Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker](https://mlsanity.com/paper/2306.00924/) (2023)
- [ChatGPT: The End of Online Exam Integrity?](https://mlsanity.com/paper/2212.09292/) (2022)
- [Template-free Prompt Tuning for Few-shot NER](https://mlsanity.com/paper/2109.13532/) (2021)
- [Evaluating Theory of Mind in Question Answering](https://mlsanity.com/paper/1808.09352/) (2018)
- [CaSiNo: A Corpus of Campsite Negotiation Dialogues for Automatic Negotiation Systems](https://mlsanity.com/paper/2103.15721/) (2021)
- [ChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relations](https://mlsanity.com/paper/2304.14827/) (2023)
- [Improving Dialog Systems for Negotiation with Personality Modeling](https://mlsanity.com/paper/2010.09954/) (2020)
- [PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Models](https://mlsanity.com/paper/2311.04044/) (2023)
- [Let's Negotiate! A Survey of Negotiation Dialogue Systems](https://mlsanity.com/paper/2212.09072/) (2022)

## Built on by (most cited first, 30 of 51)

- [MuMA-ToM: Multi-modal Multi-Agent Theory of Mind](https://mlsanity.com/paper/2408.12574/) (2024)
- [Language Models use Lookbacks to Track Beliefs](https://mlsanity.com/paper/2505.14685/) (2025)
- [Theory of Mind in Large Language Models: Assessment and Enhancement](https://mlsanity.com/paper/2505.00026/) (2025)
- [Evaluating and Enhancing LLMs Agent based on Theory of Mind in Guandan: A Multi-Player Cooperative Game under Imperfect Information](https://mlsanity.com/paper/2408.02559/) (2024)
- [ECon: On the Detection and Resolution of Evidence Conflicts](https://mlsanity.com/paper/2410.04068/) (2024)
- [PersuasiveToM: A Benchmark for Evaluating Machine Theory of Mind in Persuasive Dialogues](https://mlsanity.com/paper/2502.21017/) (2025)
- [SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactions](https://mlsanity.com/paper/2506.23046/) (2025)
- [Rethinking Theory of Mind Benchmarks for LLMs: Towards A User-Centered Perspective](https://mlsanity.com/paper/2504.10839/) (2025)
- [EgoSocialArena: Benchmarking the Social Intelligence of Large Language Models from a First-person Perspective](https://mlsanity.com/paper/2410.06195/) (2024)
- [EcomScriptBench: A Multi-task Benchmark for E-commerce Script Planning via Step-wise Intention-Driven Product Association](https://mlsanity.com/paper/2505.15196/) (2025)
- [Do Large Language Models Have a Planning Theory of Mind? Evidence from MindGames: a Multi-Step Persuasion Task](https://mlsanity.com/paper/2507.16196/) (2025)
- [The Essence of Contextual Understanding in Theory of Mind: A Study on Question Answering with Story Characters](https://mlsanity.com/paper/2501.01705/) (2025)
- [Persuasion Should be Double-Blind: A Multi-Domain Dialogue Dataset With Faithfulness Based on Causal Theory of Mind](https://mlsanity.com/paper/2502.21297/) (2025)
- [LLM-Hanabi: Evaluating Multi-Agent Gameplays with Theory-of-Mind and Rationale Inference in Imperfect Information Collaboration Game](https://mlsanity.com/paper/2510.04980/) (2025)
- [HARBOR: Exploring Persona Dynamics in Multi-Agent Competition](https://mlsanity.com/paper/2502.12149/) (2025)
- [Constrained Reasoning Chains for Enhancing Theory-of-Mind in Large Language Models](https://mlsanity.com/paper/2409.13490/) (2024)
- [Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Risks](https://mlsanity.com/paper/2505.19806/) (2025)
- [ToM-SSI: Evaluating Theory of Mind in Situated Social Interactions](https://mlsanity.com/paper/2509.05066/) (2025)
- [TactfulToM: Do LLMs Have the Theory of Mind Ability to Understand White Lies?](https://mlsanity.com/paper/2509.17054/) (2025)
- [Evaluating Multi-Turn Bargain Skills in LLM-Based Seller Agent](https://mlsanity.com/paper/2509.06341/) (2025)
- [CogToM: A Comprehensive Theory of Mind Benchmark inspired by Human Cognition for Large Language Models](https://mlsanity.com/paper/2601.15628/) (2026)
- [On the Role of Entity and Event Level Conceptualization in Generalizable Reasoning: A Survey of Tasks, Methods, Applications, and Future Directions](https://mlsanity.com/paper/2406.10885/) (2024)
- [Enhancing Conversational Agents with Theory of Mind: Aligning Beliefs, Desires, and Intentions for Human-Like Interaction](https://mlsanity.com/paper/2502.14171/) (2025)
- [XToM: Exploring the Multilingual Theory of Mind for Large Language Models](https://mlsanity.com/paper/2506.02461/) (2025)
- [Structuring the Unstructured: A Systematic Review of Text-to-Structure Generation for Agentic AI with a Universal Evaluation Framework](https://mlsanity.com/paper/2508.12257/) (2025)
- [HugAgent: A Human Simulation Benchmark for Individual-Level Reasoning](https://mlsanity.com/paper/2510.15144/) (2025)
- [To Think or Not To Think, That is The Question for Large Reasoning Models in Theory of Mind Tasks](https://mlsanity.com/paper/2602.10625/) (2026)
- [Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mind](https://mlsanity.com/paper/2602.13832/) (2026)
- [Think Thrice Before You Speak: Dual knowledge-enhanced Theory-of-Mind Reasoning for Persuasive Agents](https://mlsanity.com/paper/2605.22602/) (2026)
- [PolicyEvol-Agent: Evolving Policy via Environment Perception and Self-Awareness with Theory of Mind](https://mlsanity.com/paper/2504.15313/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have sparked substantial interest and debate concerning their potential emergence of Theory of Mind (ToM) ability. Theory of mind evaluations currently focuses on testing models using machine-generated data or game settings prone to shortcuts and spurious correlations, which lacks evaluation of machine ToM ability in real-world human interaction scenarios. This poses a pressing demand to develop new real-world scenario benchmarks. We introduce NegotiationToM, a new benchmark designed to stress-test machine ToM in real-world negotiation surrounding covered multi-dimensional mental states (i.e., desires, beliefs, and intentions). Our benchmark builds upon the Belief-Desire-Intention (BDI) agent modeling theory and conducts the necessary empirical experiments to evaluate large language models. Our findings demonstrate that NegotiationToM is challenging for state-of-the-art LLMs, as they consistently perform significantly worse than humans, even when employing the chain-of-thought (CoT) method.