mlsanityData to 7 Oct 2026

A fine-grained comparison of pragmatic language understanding in humans and language models

arXiv 2212.06801 ↗First posted 13 Dec 2022cs.CL cs.AIcited by 69

Not on the map. Why?

Abstract (arXiv)

Pragmatics and non-literal language understanding are essential to human communication, and present a long-standing challenge for artificial language models. We perform a fine-grained comparison of language models and humans on seven pragmatic phenomena, using zero-shot prompting on an expert-curated set of English materials. We ask whether models (1) select pragmatic interpretations of speaker utterances, (2) make similar error patterns as humans, and (3) use similar linguistic cues as humans to solve the tasks. We find that the largest models achieve high accuracy and match human error patterns: within incorrect responses, models favor literal interpretations over heuristic-based distractors. We also find preliminary evidence that models and humans are sensitive to similar linguistic cues. Our results suggest that pragmatic behaviors can emerge in models without explicitly constructed representations of mental states. However, models tend to struggle with phenomena relying on social expectation violations.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 26 of its 114 references that are in the map. First posted on arXiv 13 Dec 2022.

Builds on

26 of 114 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  6. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  7. 2022Galactica: A Large Language Model for Sciencecited by 736
  8. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  9. 2020Experience Grounds Languagecited by 241
  10. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  11. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  12. 2020With Little Power Comes Great Responsibilitycited by 138
14 more
  1. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 137
  2. 2022Language Models as Agent Modelscited by 131
  3. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  4. 2020Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESuppositioncited by 76
  5. 2022Testing the Ability of Language Models to Interpret Figurative Languagecited by 67
  6. 2018Evaluating Theory of Mind in Question Answeringcited by 64
  7. 2022The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMscited by 57
  8. 2021On the proper role of linguistically-oriented deep net analysis in linguistic theorizingcited by 43
  9. 2018Evaluating Compositionality in Sentence Embeddingscited by 35
  10. 2018Analyzing Compositionality-Sensitivity of NLI Modelscited by 26
  11. 2019Harnessing the linguistic signal to predict scalar inferencescited by 17
  12. 2021Concadia: Towards Image-Based Text Generation with a Purposecited by 15
  13. 2021Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Modelscited by 15
  14. 2022Just-DREAM-about-it: Figurative Language Understanding with DREAM-FLUTEcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 13; 2024: 11; 2025: 23; 2026: 21.

69 papers.

Sort:

By area:

  1. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  2. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  3. 2023Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmarkcited by 62
  4. 2024PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilitiescited by 58
  5. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 50
  6. 2023Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayingscited by 32
  7. 2023Towards A Holistic Landscape of Situated Theory of Mind in Large Language Modelscited by 32
  8. 2025Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challengescited by 31
  9. 2023Dissociating language and thought in large language modelscited by 27
  10. 2024Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformerscited by 24
  11. 2025Re-evaluating Theory of Mind evaluation in large language modelscited by 23
  12. 2024NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helpscited by 19
38 more
  1. 2023Machine Psychologycited by 16
  2. 2024Do Large Language Models Understand Conversational Implicature -- A case study with a chinese sitcomcited by 15
  3. 2023Evaluating statistical language models as pragmatic reasonerscited by 14
  4. 2024Predictions from language models for multiple-choice tasks are not robust under variation of scoring methodscited by 13
  5. 2025Triangulating LLM Progress through Benchmarks, Games, and Cognitive Testscited by 11
  6. 2025Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Modelscited by 11
  7. 2023Dealing with Semantic Underspecification in Multimodal NLPcited by 9
  8. 2025Mixture of Cognitive Reasoners: Modular Reasoning with Brain-Like Specializationcited by 9
  9. 2024How Well Do LLMs Identify Cultural Unity in Diversity?cited by 8
  10. 2025Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generationcited by 8
  11. 2022Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Modelscited by 7
  12. 2024Are LLMs good pragmatic speakers?cited by 7
  13. 2023Enhancing Interpretability using Human Similarity Judgements to Prune Word Embeddingscited by 6
  14. 2023Large Language Models are biased to overestimate profoundnesscited by 6
  15. 2023Understanding Natural Language Understanding Systems. A Critical Analysiscited by 4
  16. 2023"We Demand Justice!": Towards Social Context Grounding of Political Textscited by 4
  17. 2025Evaluating Language Models' Evaluations of Gamescited by 4
  18. 2026Post-training makes large language models less human-likecited by 4
  19. 2025They want to pretend not to understand: The Limits of Current LLMs in Interpreting Implicit Content of Political Discoursecited by 3
  20. 2026Linguistics and Human Brain: A Perspective of Computational Neurosciencecited by 3
  21. 2026Ads in AI Chatbots? An Analysis of How Large Language Models Navigate Conflicts of Interestcited by 3
  22. 2024Towards a Fully Interpretable and More Scalable RSA Model for Metaphor Understandingcited by 2
  23. 2025Leveraging Human Production-Interpretation Asymmetries to Test LLM Cognitive Plausibilitycited by 2
  24. 2025Playpen: An Environment for Exploring Learning Through Conversational Interactioncited by 2
  25. 2025How important is language for human-like intelligence?cited by 2
  26. 2024Emotional RAG LLMs: Reading Comprehension for the Open Internetcited by 1
  27. 2025Generative AI and the future of scientometrics: current topics and future questionscited by 1
  28. 2025QSTN: A Modular Framework for Robust Questionnaire Inference with Large Language Modelscited by 1
  29. 2026Can LLMs interpret figurative language as humans do?: surface-level vs representational similaritycited by 1
  30. 2026How Hypocritical Is Your LLM judge? Listener-Speaker Asymmetries in the Pragmatic Competence of Large Language Modelscited by 1
  31. 2026Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responsescited by 1
  32. 2026Consistency Without Alignment: Item-Sensitive Language Models Indistinguishable From Randomcited by 1
  33. 2024FAC2E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognitioncited by 0
  34. 2024Do language models capture implied discourse meanings? An investigation with exhaustivity implicatures of Korean morphologycited by 0
  35. 2025Evidence of conceptual mastery in the application of rules by Large Language Modelscited by 0
  36. 2025ChatGPT for President! Presupposed content in politicians versus GPT-generated textscited by 0
  37. 2025Using Perspectival Words Is Harder Than Vocabulary Words for Humans and Even More So for Multimodal Language Modelscited by 0
  38. 2025(RSA)2: A Rhetorical-Strategy-Aware Rational Speech Act Framework for Figurative Language Understandingcited by 0

50 most cited of 69 All 69 (JSON)