# "No, they did not": Dialogue response dynamics in pre-trained language models

arXiv: https://arxiv.org/abs/2210.02526 · HTML: https://mlsanity.com/paper/2210.02526/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 5 Oct 2022.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [PAWS: Paraphrase Adversaries from Word Scrambling](https://mlsanity.com/paper/1904.01130/) (2019)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://mlsanity.com/paper/1608.04207/) (2016)
- [Assessing BERT's Syntactic Abilities](https://mlsanity.com/paper/1901.05287/) (2019)
- [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019)
- [TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue](https://mlsanity.com/paper/2004.06871/) (2020)
- [Dissecting Contextual Word Embeddings: Architecture and Representation](https://mlsanity.com/paper/1808.08949/) (2018)
- [PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable](https://mlsanity.com/paper/1910.07931/) (2019)
- [ConveRT: Efficient and Accurate Conversational Representations from Transformers](https://mlsanity.com/paper/1911.03688/) (2019)
- [minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models](https://mlsanity.com/paper/2203.13112/) (2022)
- [Topic-Driven and Knowledge-Aware Transformer for Dialogue Emotion Detection](https://mlsanity.com/paper/2106.01071/) (2021)
- [Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems](https://mlsanity.com/paper/1907.01166/) (2019)
- [SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity](https://mlsanity.com/paper/1608.00869/) (2016)
- [Probing What Different NLP Tasks Teach Machines about Function Word Comprehension](https://mlsanity.com/paper/1904.11544/) (2019)
- [Discourse Probing of Pretrained Language Models](https://mlsanity.com/paper/2104.05882/) (2021)
- [DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances](https://mlsanity.com/paper/2012.01775/) (2020)
- [Assessing Composition in Sentence Vector Representations](https://mlsanity.com/paper/1809.03992/) (2018)
- [Pragmatic competence of pre-trained language models through the lens of discourse connectives](https://mlsanity.com/paper/2109.12951/) (2021)
- [Probing Multilingual Language Models for Discourse](https://mlsanity.com/paper/2106.04832/) (2021)
- [Does BERT agree? Evaluating knowledge of structure dependence through agreement relations](https://mlsanity.com/paper/1908.09892/) (2019)

## Built on by (most cited first, 1 of 1)

- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)

## Abstract (arXiv)

A critical component of competence in language is being able to identify relevant components of an utterance and reply appropriately. In this paper we examine the extent of such dialogue response sensitivity in pre-trained language models, conducting a series of experiments with a particular focus on sensitivity to dynamics involving phenomena of at-issueness and ellipsis. We find that models show clear sensitivity to a distinctive role of embedded clauses, and a general preference for responses that target main clause content of prior utterances. However, the results indicate mixed and generally weak trends with respect to capturing the full range of dynamics involved in targeting at-issue versus not-at-issue content. Additionally, models show fundamental limitations in grasp of the dynamics governing ellipsis, and response selections show clear interference from superficial factors that outweigh the influence of principled discourse constraints.