# DialAug: Mixing up Dialogue Contexts in Contrastive Learning for Robust Conversational Modeling

arXiv: https://arxiv.org/abs/2204.07679 · HTML: https://mlsanity.com/paper/2204.07679/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 15 Apr 2022.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019)
- [CLEAR: Contrastive Learning for Sentence Representation](https://mlsanity.com/paper/2012.15466/) (2020)
- [TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue](https://mlsanity.com/paper/2004.06871/) (2020)
- [The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems](https://mlsanity.com/paper/1506.08909/) (2015)
- [CERT: Contrastive Self-supervised Learning for Language Understanding](https://mlsanity.com/paper/2005.12766/) (2020)
- [Certified Robustness to Adversarial Word Substitutions](https://mlsanity.com/paper/1909.00986/) (2019)
- [Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring](https://mlsanity.com/paper/1905.01969/) (2019)
- [Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset](https://mlsanity.com/paper/1909.05358/) (2019)
- [Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots](https://mlsanity.com/paper/2004.03588/) (2020)
- [A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation](https://mlsanity.com/paper/2009.13818/) (2020)
- [PyTorch Metric Learning](https://mlsanity.com/paper/2008.09164/) (2020)
- [Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation](https://mlsanity.com/paper/2010.12836/) (2020)
- [Training Neural Response Selection for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1906.01543/) (2019)
- [How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?](https://mlsanity.com/paper/2010.01764/) (2020)
- [Pretraining Methods for Dialog Context Representation Learning](https://mlsanity.com/paper/1906.00414/) (2019)
- [Few-Shot Text Classification with Triplet Networks, Data Augmentation, and Curriculum Learning](https://mlsanity.com/paper/2103.07552/) (2021)
- [Learning an Effective Context-Response Matching Model with Self-Supervised Tasks for Retrieval-based Dialogues](https://mlsanity.com/paper/2009.06265/) (2020)
- [NAT: Noise-Aware Training for Robust Neural Sequence Labeling](https://mlsanity.com/paper/2005.07162/) (2020)
- [Self-Supervised Contrastive Learning for Efficient User Satisfaction Prediction in Conversational Agents](https://mlsanity.com/paper/2010.11230/) (2020)
- [Contrastive Fine-tuning Improves Robustness for Neural Rankers](https://mlsanity.com/paper/2105.12932/) (2021)

## Built on by (most cited first, 3 of 3)

- [Data Augmentation for Conversational AI](https://mlsanity.com/paper/2309.04739/) (2023)
- [Improving the Robustness of Knowledge-Grounded Dialogue via Contrastive Learning](https://mlsanity.com/paper/2401.04361/) (2024)
- [Deploying a Retrieval based Response Model for Task Oriented Dialogues](https://mlsanity.com/paper/2210.14379/) (2022)

## Abstract (arXiv)

Retrieval-based conversational systems learn to rank response candidates for a given dialogue context by computing the similarity between their vector representations. However, training on a single textual form of the multi-turn context limits the ability of a model to learn representations that generalize to natural perturbations seen during inference. In this paper we propose a framework that incorporates augmented versions of a dialogue context into the learning objective. We utilize contrastive learning as an auxiliary objective to learn robust dialogue context representations that are invariant to perturbations injected through the augmentation method. We experiment with four benchmark dialogue datasets and demonstrate that our framework combines well with existing augmentation methods and can significantly improve over baseline BERT-based ranking architectures. Furthermore, we propose a novel data augmentation method, ConMix, that adds token level perturbations through stochastic mixing of tokens from other contexts in the batch. We show that our proposed augmentation method outperforms previous data augmentation approaches, and provides dialogue representations that are more robust to common perturbations seen during inference.