# DiPlomat: A Dialogue Dataset for Situated Pragmatic Reasoning

arXiv: https://arxiv.org/abs/2306.09030 · HTML: https://mlsanity.com/paper/2306.09030/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 4 of its 69 references that are in the map.
- First posted on arXiv 15 Jun 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 4 of 4)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Improving alignment of dialogue agents via targeted human judgements](https://mlsanity.com/paper/2209.14375/) (2022)

## Built on by (most cited first, 14 of 14)

- [AutoMix: Automatically Mixing Language Models](https://mlsanity.com/paper/2310.12963/) (2023)
- [Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges](https://mlsanity.com/paper/2502.12378/) (2025)
- [The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trends](https://mlsanity.com/paper/2409.14195/) (2024)
- [Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning](https://mlsanity.com/paper/2506.08354/) (2025)
- [Is There Any Social Principle for LLM-Based Agents?](https://mlsanity.com/paper/2308.11136/) (2023)
- [Affective Visual Dialog: A Large-Scale Benchmark for Emotional Reasoning Based on Visually Grounded Conversations](https://mlsanity.com/paper/2308.16349/) (2023)
- [Are they lovers or friends? Evaluating LLMs' Social Reasoning in English and Korean Dialogues](https://mlsanity.com/paper/2510.19028/) (2025)
- [Pragmatic Theories Enhance Understanding of Implied Meanings in LLMs](https://mlsanity.com/paper/2510.26253/) (2025)
- [BEDA: Belief Estimation as Probabilistic Constraints for Performing Strategic Dialogue Acts](https://mlsanity.com/paper/2512.24885/) (2025)
- [Surfacing the Unsaid: CUE-Bench for Affective Stance in Chinese Discourse](https://mlsanity.com/paper/2608.10810/) (2026)
- [CEI: A Benchmark for Evaluating Pragmatic Reasoning in Language Models](https://mlsanity.com/paper/2603.09993/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)
- [Leveraging Speech Acts for Low-Data and Cross-Domain Conversation Derailment Forecasting](https://mlsanity.com/paper/2608.25359/) (2026)
- [The Shape of Power: A Multilingual Framework for Social Power Reasoning in Dialogues](https://mlsanity.com/paper/2608.28144/) (2026)

## Abstract (arXiv)

Pragmatic reasoning plays a pivotal role in deciphering implicit meanings that frequently arise in real-life conversations and is essential for the development of communicative social agents. In this paper, we introduce a novel challenge, DiPlomat, aiming at benchmarking machines' capabilities on pragmatic reasoning and situated conversational understanding. Compared with previous works that treat different figurative expressions (e.g. metaphor, sarcasm) as individual tasks, DiPlomat provides a cohesive framework towards general pragmatic understanding. Our dataset is created through the utilization of Amazon Mechanical Turk ( AMT ), resulting in a total of 4, 177 multi-turn dialogues. In conjunction with the dataset, we propose two tasks, Pragmatic Identification and Reasoning (PIR) and Conversational Question Answering (CQA). Experimental results with state-of-the-art (SOTA) neural architectures reveal several significant findings: 1) large language models ( LLMs) exhibit poor performance in tackling this subjective domain; 2) comprehensive comprehension of context emerges as a critical factor for establishing benign human-machine interactions; 3) current models defect in the application of pragmatic reasoning. As a result, we call on more attention to improve the ability of context understanding, reasoning, and implied meaning modeling.