# Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms

arXiv: https://arxiv.org/abs/2609.29001 · HTML: https://mlsanity.com/paper/2609.29001/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 37 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 14 of 14)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [A Computational Approach to Politeness with Application to Social Factors](https://mlsanity.com/paper/1306.6078/) (2013)
- [PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities](https://mlsanity.com/paper/2401.07078/) (2024)
- [The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs](https://mlsanity.com/paper/2210.14986/) (2022)
- [Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges](https://mlsanity.com/paper/2502.12378/) (2025)
- [Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach](https://mlsanity.com/paper/2503.21819/) (2025)
- [Comparing human and LLM politeness strategies in free production](https://mlsanity.com/paper/2506.09391/) (2025)
- [Computational Politeness in Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.12814/) (2024)
- [Assessing Social Alignment: Do Personality-Prompted Large Language Models Behave Like Humans?](https://mlsanity.com/paper/2412.16772/) (2024)
- [To Mask or to Mirror: Human-AI Alignment in Collective Reasoning](https://mlsanity.com/paper/2510.01924/) (2025)
- [Evaluating AI Alignment in LLMs: Output Analysis of Value Priorities Across 75 Models with Human Benchmarking](https://mlsanity.com/paper/2506.12617/) (2025)

## Abstract (arXiv)

Despite strong performance on standard benchmarks, it remains unclear whether large language models (LLMs) evaluate social pragmatics in ways that align with human judgments. We evaluate LLM politeness judgments using two English-language datasets with complementary annotation formats: continuous human ratings and three-way categorical labels. Across the seven evaluated models, we find that inter-model agreement is stronger than model--human agreement. Strategy-level analyses suggest that model--human alignment is associated with explicit linguistic cues, while some rapport-building strategies occur more frequently in misaligned cases. In the categorical task, model predictions exhibit systematic neutral compression, characterized by the overproduction of Neutral labels and the underprediction of Impolite labels. This pattern persists when expert consensus is used as the reference on a diagnostic subset. Our findings highlight the need for pragmatic evaluations that go beyond aggregate agreement metrics by examining directional patterns of model--human disagreement across different human references.