mlsanityData to 14 Oct 2026

The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs

arXiv 2210.14986 ↗First posted 26 Oct 2022cs.CLcited by 58

Not on the map. Why?

Abstract (arXiv)

Despite widespread use of LLMs as conversational agents, evaluations of performance fail to capture a crucial aspect of communication: interpreting language in context -- incorporating its pragmatics. Humans interpret language using beliefs and prior knowledge about the world. For example, we intuitively understand the response "I wore gloves" to the question "Did you leave fingerprints?" as meaning "No". To investigate whether LLMs have the ability to make this type of inference, known as an implicature, we design a simple task and evaluate four categories of widely used state-of-the-art models. We find that, despite only evaluating on utterances that require a binary inference (yes or no), models in three of these categories perform close to random. However, LLMs instruction-tuned at the example-level perform significantly better. These results suggest that certain fine-tuning strategies are far better at inducing pragmatic understanding in models. We present our findings as the starting point for further research into evaluating how LLMs interpret language in context and to drive the development of more pragmatic and useful models of human discourse.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Oct 2022.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 8; 2024: 15; 2025: 16; 2026: 16.

58 papers.

Sort:

By area:

  1. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  2. 2025Agentic Large Language Models, a surveycited by 108
  3. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  4. 2022A fine-grained comparison of pragmatic language understanding in humans and language modelscited by 70
  5. 2023Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelscited by 64
  6. 2024PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilitiescited by 59
  7. 2024Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMscited by 59
  8. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
  9. 2023Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayingscited by 32
  10. 2025Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challengescited by 31
  11. 2024(Ir)rationality and Cognitive Biases in Large Language Modelscited by 23
  12. 2025Re-evaluating Theory of Mind evaluation in large language modelscited by 23
38 more
  1. 2023Machine Psychologycited by 22
  2. 2023Learning to Reason over Scene Graphs: A Case Study of Finetuning GPT-2 into a Robot Language Model for Grounded Task Planningcited by 16
  3. 2023Evaluating statistical language models as pragmatic reasonerscited by 15
  4. 2022Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approachescited by 12
  5. 2023Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Modelscited by 12
  6. 2024The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trendscited by 11
  7. 2025Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Modelscited by 11
  8. 2025Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questionscited by 10
  9. 2023Dealing with Semantic Underspecification in Multimodal NLPcited by 9
  10. 2025Non-literal Understanding of Number Words by Language Modelscited by 9
  11. 2025Evaluation and Facilitation of Online Discussions in the LLM Era: A Surveycited by 9
  12. 2025Pragmatic Inference Chain (PIC) Improving LLMs' Reasoning of Authentic Implicit Toxic Languagecited by 8
  13. 2024LocalValueBench: A Collaboratively Built and Extensible Benchmark for Evaluating Localized Value Alignment and Ethical Safety in Large Language Modelscited by 7
  14. 2024Are LLMs good pragmatic speakers?cited by 7
  15. 2025Towards Robust ESG Analysis Against Greenwashing Risks: Aspect-Action Analysis with Cross-Category Generalizationcited by 7
  16. 2024Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmaticscited by 3
  17. 2025The Pragmatic Mind of Machines: Tracing the Emergence of Pragmatic Competence in Large Language Modelscited by 3
  18. 2025They want to pretend not to understand: The Limits of Current LLMs in Interpreting Implicit Content of Political Discoursecited by 3
  19. 2025From Training to Generalization: Improving Moral Reasoning Through Pragmatic Inferencecited by 2
  20. 2025Pragmatic Theories Enhance Understanding of Implied Meanings in LLMscited by 2
  21. 2026They Infer What You Meant: Models Represent Communicative Intent More Reliably Than They Act On Itcited by 2
  22. 2024Can AI and humans genuinely communicate?cited by 1
  23. 2024Is the Pope Catholic? Yes, the Pope is Catholic. Generative Evaluation of Non-Literal Intent Resolution in LLMscited by 1
  24. 2024Emotional RAG LLMs: Reading Comprehension for the Open Internetcited by 1
  25. 2024Parameter Choice and Neuro-Symbolic Approaches for Deep Domain-Invariant Learningcited by 1
  26. 2025Integrating Neural and Symbolic Components in a Model of Pragmatic Question-Answeringcited by 1
  27. 2026Unveiling the Limits of Large Language Models in Inferring Pragmatic Meaning from Non-Verbal Responsescited by 1
  28. 2024Are Large Language Models Aligned with People's Social Intuitions for Human-Robot Interactions?cited by 0
  29. 2024Towards an Analysis of Discourse and Interactional Pragmatic Reasoning Capabilities of Large Language Modelscited by 0
  30. 2024Generative AI, Pragmatics, and Authenticity in Second Language Learningcited by 0
  31. 2025ChatGPT for President! Presupposed content in politicians versus GPT-generated textscited by 0
  32. 2025Implicature in Interaction: Understanding Implicature Improves Alignment in Human-LLM Interactioncited by 0
  33. 2026CEI: A Benchmark for Evaluating Pragmatic Reasoning in Language Modelscited by 0
  34. 2026When Contextual Inference Fails: Cancelability in Interactive Instruction Followingcited by 0
  35. 2026Profiling learners' affective engagement: Emotion AI, intercultural pragmatics, and language learningcited by 0
  36. 2026Continuous Interpretive Steering for Scalar Diversitycited by 0
  37. 2026Do LLMs Use Cultural Knowledge Without Being Told? A Multilingual Evaluation of Implicit Pragmatic Adaptationcited by 0
  38. 2026DRInQ: Evaluating Conversational Implicature with Controlled Context Variationcited by 0

50 most cited of 58 All 58 (JSON)