# Expanding the Set of Pragmatic Considerations in Conversational AI

arXiv: https://arxiv.org/abs/2310.18435 · HTML: https://mlsanity.com/paper/2310.18435/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 83 references that are in the map.
- First posted on arXiv 27 Oct 2023.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage](https://mlsanity.com/paper/2208.03188/) (2022)
- [Language models show human-like content effects on reasoning tasks](https://mlsanity.com/paper/2207.07051/) (2022)
- [Re-evaluating Evaluation in Text Summarization](https://mlsanity.com/paper/2010.07100/) (2020)
- [MIME: MIMicking Emotions for Empathetic Response Generation](https://mlsanity.com/paper/2010.01454/) (2020)
- [Proactive Human-Machine Conversation with Explicit Conversation Goals](https://mlsanity.com/paper/1906.05572/) (2019)
- [Language Model Evaluation Beyond Perplexity](https://mlsanity.com/paper/2106.00085/) (2021)
- [Prompted LLMs as Chatbot Modules for Long Open-domain Conversation](https://mlsanity.com/paper/2305.04533/) (2023)
- [Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes](https://mlsanity.com/paper/2109.08828/) (2021)
- [Ask what's missing and what's useful: Improving Clarification Question Generation using Global Knowledge](https://mlsanity.com/paper/2104.06828/) (2021)
- [Pragmatic competence of pre-trained language models through the lens of discourse connectives](https://mlsanity.com/paper/2109.12951/) (2021)
- [Pragmatic Issue-Sensitive Image Captioning](https://mlsanity.com/paper/2004.14451/) (2020)
- [Learning to Mediate Disparities Towards Pragmatic Communication](https://mlsanity.com/paper/2203.13685/) (2022)
- [A recipe for annotating grounded clarifications](https://mlsanity.com/paper/2104.08964/) (2021)
- [Calibrate your listeners! Robust communication-based training for pragmatic speakers](https://mlsanity.com/paper/2110.05422/) (2021)
- [Long-form analogies generated by chatGPT lack human-like psycholinguistic properties](https://mlsanity.com/paper/2306.04537/) (2023)
- [Improving Social Meaning Detection with Pragmatic Masking and Surrogate Fine-Tuning](https://mlsanity.com/paper/2108.00356/) (2021)
- [Deciding Whether to Ask Clarifying Questions in Large-Scale Spoken Language Understanding](https://mlsanity.com/paper/2109.12451/) (2021)

## Abstract (arXiv)

Despite considerable performance improvements, current conversational AI systems often fail to meet user expectations. We discuss several pragmatic limitations of current conversational AI systems. We illustrate pragmatic limitations with examples that are syntactically appropriate, but have clear pragmatic deficiencies. We label our complaints as "Turing Test Triggers" (TTTs) as they indicate where current conversational AI systems fall short compared to human behavior. We develop a taxonomy of pragmatic considerations intended to identify what pragmatic competencies a conversational AI system requires and discuss implications for the design and evaluation of conversational AI systems.