# Does GPT-4 surpass human performance in linguistic pragmatics?

arXiv: https://arxiv.org/abs/2312.09545 · HTML: https://mlsanity.com/paper/2312.09545/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 48 references that are in the map.
- First posted on arXiv 15 Dec 2023.

## Builds on (most cited first, 8 of 8)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation](https://mlsanity.com/paper/2301.01768/) (2023)
- [Moral Foundations of Large Language Models](https://mlsanity.com/paper/2310.15337/) (2023)
- [Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identity](https://mlsanity.com/paper/2209.12106/) (2022)
- [Exploring The Design of Prompts For Applying GPT-3 based Chatbots: A Mental Wellbeing Case Study on Mechanical Turk](https://mlsanity.com/paper/2209.11344/) (2022)
- [Cross-Platform Social Dynamics: An Analysis of ChatGPT and COVID-19 Vaccine Conversations](https://mlsanity.com/paper/2310.11116/) (2023)
- [CERN for AI: A Theoretical Framework for Autonomous Simulation-Based Artificial Intelligence Testing and Alignment](https://mlsanity.com/paper/2312.09402/) (2023)

## Built on by (most cited first, 16 of 16)

- [GPT-4 vs. Human Translators: A Comprehensive Evaluation of Translation Quality Across Languages, Domains, and Expertise Levels](https://mlsanity.com/paper/2407.03658/) (2024)
- [MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models](https://mlsanity.com/paper/2406.07736/) (2024)
- [Benchmarking GPT-4 against Human Translators: A Comprehensive Evaluation Across Languages, Domains, and Expertise Levels](https://mlsanity.com/paper/2411.13775/) (2024)
- [Pragmatic Competence Evaluation of Large Language Models for the Korean Language](https://mlsanity.com/paper/2403.12675/) (2024)
- [CONFLARE: CONFormal LArge language model REtrieval](https://mlsanity.com/paper/2404.04287/) (2024)
- [Psychological and behavioural responses in human-agent vs. human-human interactions: a systematic review and meta-analysis](https://mlsanity.com/paper/2509.21542/) (2025)
- [Assessing Algorithmic Bias in Language-Based Depression Detection: A Comparison of DNN and LLM Approaches](https://mlsanity.com/paper/2509.25795/) (2025)
- [Evaluating Large Language Models Against Human Annotators in Latent Content Analysis: Sentiment, Political Leaning, Emotional Intensity, and Sarcasm](https://mlsanity.com/paper/2501.02532/) (2025)
- [Representational Similarity Alignment of Figurative Language in Humans and Large Language Models](https://mlsanity.com/paper/2601.09041/) (2026)
- [From Human-Level AI Tales to AI Leveling Human Scales](https://mlsanity.com/paper/2602.18911/) (2026)
- [LLM Agents Predict Social Media Reactions but Do Not Outperform Text Classifiers: Benchmarking Simulation Accuracy Using 120K+ Personas of 1511 Humans](https://mlsanity.com/paper/2604.19787/) (2026)
- [Pragmatics beyond humans: meaning, communication, and LLMs](https://mlsanity.com/paper/2508.06167/) (2025)
- [Implicature in Interaction: Understanding Implicature Improves Alignment in Human-LLM Interaction](https://mlsanity.com/paper/2510.25426/) (2025)
- [Psychological Influences of Conversational AI: Research and Design Directions for Reducing Harm and Promoting Well-Being](https://mlsanity.com/paper/2607.25057/) (2026)
- [Thinking Less to Simulate Better: Intuitive Prompting Improves LLM Agents Simulating Individual Social Media Reactions, Including Unfamiliar Content](https://mlsanity.com/paper/2609.30563/) (2026)
- [Beyond End-to-End Black Box Mapping: An Intentional Agent Framework for Cognitive-driven Facial Reaction Generation](https://mlsanity.com/paper/2609.34419/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) become increasingly integrated into everyday life as general purpose multimodal AI systems, their capabilities to simulate human understanding are under examination. This study investigates LLMs ability to interpret linguistic pragmatics, which involves context and implied meanings. Using Grice communication principles, we evaluated both LLMs (GPT-2, GPT-3, GPT-3.5, GPT-4, and Bard) and human subjects (N = 147) on dialogue-based tasks. Human participants included 71 primarily Serbian students and 76 native English speakers from the United States. Findings revealed that LLMs, particularly GPT-4, outperformed humans. GPT4 achieved the highest score of 4.80, surpassing the best human score of 4.55. Other LLMs performed well: GPT 3.5 scored 4.10, Bard 3.75, and GPT-3 3.25. GPT-2 had the lowest score of 1.05. The average LLM score was 3.39, exceeding the human cohorts averages of 2.80 (Serbian students) and 2.34 (U.S. participants). In the ranking of all 155 subjects (including LLMs and humans), GPT-4 secured the top position, while the best human ranked second. These results highlight significant progress in LLMs ability to simulate understanding of linguistic pragmatics. Future studies should confirm these findings with more dialogue-based tasks and diverse participants. This research has important implications for advancing general-purpose AI models in various communication-centered tasks, including potential application in humanoid robots in the future.