# DialogueCSE: Dialogue-based Contrastive Learning of Sentence Embeddings

arXiv: https://arxiv.org/abs/2109.12599 · HTML: https://mlsanity.com/paper/2109.12599/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 37 references that are in the map.
- First posted on arXiv 26 Sep 2021.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [Efficient Natural Language Response Suggestion for Smart Reply](https://mlsanity.com/paper/1705.00652/) (2017)
- [Whitening Sentence Representations for Better Semantics and Faster Retrieval](https://mlsanity.com/paper/2103.15316/) (2021)
- [An efficient framework for learning sentence representations](https://mlsanity.com/paper/1803.02893/) (2018)
- [CLEAR: Contrastive Learning for Sentence Representation](https://mlsanity.com/paper/2012.15466/) (2020)
- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://mlsanity.com/paper/2102.08473/) (2021)
- [ConveRT: Efficient and Accurate Conversational Representations from Transformers](https://mlsanity.com/paper/1911.03688/) (2019)
- [Learning Distributed Representations of Sentences from Unlabelled Data](https://mlsanity.com/paper/1602.03483/) (2016)
- [An Unsupervised Sentence Embedding Method by Mutual Information Maximization](https://mlsanity.com/paper/2009.12061/) (2020)
- [On the Sentence Embeddings from Pre-trained Language Models](https://mlsanity.com/paper/2011.05864/) (2020)
- [Modeling Multi-turn Conversation with Deep Utterance Aggregation](https://mlsanity.com/paper/1806.09102/) (2018)
- [Microsoft Dialogue Challenge: Building End-to-End Task-Completion Dialogue Systems](https://mlsanity.com/paper/1807.11125/) (2018)
- [The JDDC Corpus: A Large-Scale Multi-Turn Chinese Dialogue Dataset for E-commerce Customer Service](https://mlsanity.com/paper/1911.09969/) (2019)
- [Learning Semantic Textual Similarity from Conversations](https://mlsanity.com/paper/1804.07754/) (2018)
- [Conversational Contextual Cues: The Case of Personalization and History for Response Ranking](https://mlsanity.com/paper/1606.00372/) (2016)

## Built on by (most cited first, 19 of 19)

- [Same Author or Just Same Topic? Towards Content-Independent Style Representations](https://mlsanity.com/paper/2204.04907/) (2022)
- [Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systems](https://mlsanity.com/paper/2205.15060/) (2022)
- [Adaptive Contrastive Learning on Multimodal Transformer for Review Helpfulness Predictions](https://mlsanity.com/paper/2211.03524/) (2022)
- [Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2209.09433/) (2022)
- [Dialog2Flow: Pre-training Soft-Contrastive Action-Driven Sentence Embeddings for Automatic Dialog Flow Extraction](https://mlsanity.com/paper/2410.18481/) (2024)
- [Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders](https://mlsanity.com/paper/2205.00267/) (2022)
- [Coarse-to-Fine Knowledge Selection for Document Grounded Dialogs](https://mlsanity.com/paper/2302.11849/) (2023)
- [Efficient Domain Adaptation of Sentence Embeddings Using Adapters](https://mlsanity.com/paper/2307.03104/) (2023)
- [DivTOD: Unleashing the Power of LLMs for Diversifying Task-Oriented Dialogue Representations](https://mlsanity.com/paper/2404.00557/) (2024)
- [Layout-Aware Information Extraction for Document-Grounded Dialogue: Dataset, Method and Demonstration](https://mlsanity.com/paper/2207.06717/) (2022)
- [Learning Interpretable Latent Dialogue Actions With Less Supervision](https://mlsanity.com/paper/2209.11128/) (2022)
- [Dial2vec: Self-Guided Contrastive Learning of Unsupervised Dialogue Embeddings](https://mlsanity.com/paper/2210.15332/) (2022)
- [Imagination is All You Need! Curved Contrastive Learning for Abstract Sequence Modeling Utilized on Long Short-Term Dialogue Planning](https://mlsanity.com/paper/2211.07591/) (2022)
- [Template-assisted Contrastive Learning of Task-oriented Dialogue Sentence Embeddings](https://mlsanity.com/paper/2305.14299/) (2023)
- [FutureTOD: Teaching Future Knowledge to Pre-trained Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2306.10315/) (2023)
- [Accuracy on In-Domain Samples Matters When Building Out-of-Domain detectors: A Reply to Marek et al. (2021)](https://mlsanity.com/paper/2205.11887/) (2022)
- [SCAT: Robust Self-supervised Contrastive Learning via Adversarial Training for Text Classification](https://mlsanity.com/paper/2307.01488/) (2023)
- [Contrastive Learning for Inference in Dialogue](https://mlsanity.com/paper/2310.12467/) (2023)
- [BootTOD: Bootstrap Task-oriented Dialogue Representations by Aligning Diverse Responses](https://mlsanity.com/paper/2403.01163/) (2024)

## Abstract (arXiv)

Learning sentence embeddings from dialogues has drawn increasing attention due to its low annotation cost and high domain adaptability. Conventional approaches employ the siamese-network for this task, which obtains the sentence embeddings through modeling the context-response semantic relevance by applying a feed-forward network on top of the sentence encoders. However, as the semantic textual similarity is commonly measured through the element-wise distance metrics (e.g. cosine and L2 distance), such architecture yields a large gap between training and evaluating. In this paper, we propose DialogueCSE, a dialogue-based contrastive learning approach to tackle this issue. DialogueCSE first introduces a novel matching-guided embedding (MGE) mechanism, which generates a context-aware embedding for each candidate response embedding (i.e. the context-free embedding) according to the guidance of the multi-turn context-response matching matrices. Then it pairs each context-aware embedding with its corresponding context-free embedding and finally minimizes the contrastive loss across all pairs. We evaluate our model on three multi-turn dialogue datasets: the Microsoft Dialogue Corpus, the Jing Dong Dialogue Corpus, and the E-commerce Dialogue Corpus. Evaluation results show that our approach significantly outperforms the baselines across all three datasets in terms of MAP and Spearman's correlation measures, demonstrating its effectiveness. Further quantitative experiments show that our approach achieves better performance when leveraging more dialogue context and remains robust when less training data is provided.