# Learning an Effective Context-Response Matching Model with Self-Supervised Tasks for Retrieval-based Dialogues

arXiv: https://arxiv.org/abs/2009.06265 · HTML: https://mlsanity.com/paper/2009.06265/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Sep 2020.

## Built on by (most cited first, 30 of 34)

- [InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning](https://mlsanity.com/paper/2205.12673/) (2022)
- [Structural Pre-training for Dialogue Comprehension](https://mlsanity.com/paper/2105.10956/) (2021)
- [Leveraging Large Language Models for Automated Dialogue Analysis](https://mlsanity.com/paper/2309.06490/) (2023)
- [Advances in Multi-turn Dialogue Comprehension: A Survey](https://mlsanity.com/paper/2103.03125/) (2021)
- [Exploring Dense Retrieval for Dialogue Response Selection](https://mlsanity.com/paper/2110.06612/) (2021)
- [Dialogue-oriented Pre-training](https://mlsanity.com/paper/2106.00420/) (2021)
- [Contextual Fine-to-Coarse Distillation for Coarse-grained Response Selection in Open-Domain Conversations](https://mlsanity.com/paper/2109.13087/) (2021)
- [STS-CCL: Spatial-Temporal Synchronous Contextual Contrastive Learning for Urban Traffic Forecasting](https://mlsanity.com/paper/2307.02507/) (2023)
- [Small Changes Make Big Differences: Improving Multi-turn Response Selection in Dialogue Systems via Fine-Grained Contrastive Learning](https://mlsanity.com/paper/2111.10154/) (2021)
- [A Systematic Evaluation of Response Selection for Open Domain Dialogue](https://mlsanity.com/paper/2208.04379/) (2022)
- [Bridging the Gap between Language Model and Reading Comprehension: Unsupervised MRC via Self-Supervision](https://mlsanity.com/paper/2107.08582/) (2021)
- [DialAug: Mixing up Dialogue Contexts in Contrastive Learning for Robust Conversational Modeling](https://mlsanity.com/paper/2204.07679/) (2022)
- [Taxonomy of Abstractive Dialogue Summarization: Scenarios, Approaches and Future Directions](https://mlsanity.com/paper/2210.09894/) (2022)
- [Uni-Encoder: A Fast and Accurate Response Selection Paradigm for Generation-Based Dialogue Systems](https://mlsanity.com/paper/2106.01263/) (2021)
- [WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue](https://mlsanity.com/paper/2108.01487/) (2021)
- [Re-entry Prediction for Online Conversations via Self-Supervised Learning](https://mlsanity.com/paper/2109.02020/) (2021)
- [Two-Level Supervised Contrastive Learning for Response Selection in Multi-Turn Dialogue](https://mlsanity.com/paper/2203.00793/) (2022)
- [Dial-MAE: ConTextual Masked Auto-Encoder for Retrieval-based Dialogue Systems](https://mlsanity.com/paper/2306.04357/) (2023)
- [An Evaluation Dataset and Strategy for Building Robust Multi-turn Response Selection Model](https://mlsanity.com/paper/2109.04834/) (2021)
- [CORAL: Contextual Response Retrievability Loss Function for Training Dialog Generation Models](https://mlsanity.com/paper/2205.10558/) (2022)
- [Learning Locality and Isotropy in Dialogue Modeling](https://mlsanity.com/paper/2205.14583/) (2022)
- [Deploying a Retrieval based Response Model for Task Oriented Dialogues](https://mlsanity.com/paper/2210.14379/) (2022)
- [Tri-Attention: Explicit Context-Aware Attention Mechanism for Natural Language Processing](https://mlsanity.com/paper/2211.02899/) (2022)
- [Intent-calibrated Self-training for Answer Selection in Open-domain Dialogues](https://mlsanity.com/paper/2307.06703/) (2023)
- [A Diverse and Effective Retrieval-Based Debt Collection System with Expert Knowledge](https://mlsanity.com/paper/2504.06273/) (2025)
- [Chat-Ghosting: A Comparative Study of Methods for Auto-Completion in Dialog Systems](https://mlsanity.com/paper/2507.05940/) (2025)
- [Advances in Multi-turn Dialogue Comprehension: A Survey](https://mlsanity.com/paper/2110.04984/) (2021)
- [Mix-and-Match: Scalable Dialog Response Retrieval using Gaussian Mixture Embeddings](https://mlsanity.com/paper/2204.02710/) (2022)
- [Sparse and Dense Approaches for the Full-rank Retrieval of Responses for Dialogues](https://mlsanity.com/paper/2204.10558/) (2022)
- [Causal Inference for Chatting Handoff](https://mlsanity.com/paper/2210.02862/) (2022)

## Abstract (arXiv)

Building an intelligent dialogue system with the ability to select a proper response according to a multi-turn context is a great challenging task. Existing studies focus on building a context-response matching model with various neural architectures or PLMs and typically learning with a single response prediction task. These approaches overlook many potential training signals contained in dialogue data, which might be beneficial for context understanding and produce better features for response prediction. Besides, the response retrieved from existing dialogue systems supervised by the conventional way still faces some critical challenges, including incoherence and inconsistency. To address these issues, in this paper, we propose learning a context-response matching model with auxiliary self-supervised tasks designed for the dialogue data based on pre-trained language models. Specifically, we introduce four self-supervised tasks including next session prediction, utterance restoration, incoherence detection and consistency discrimination, and jointly train the PLM-based response selection model with these auxiliary tasks in a multi-task manner. By this means, the auxiliary tasks can guide the learning of the matching model to achieve a better local optimum and select a more proper response. Experiment results on two benchmarks indicate that the proposed auxiliary self-supervised tasks bring significant improvement for multi-turn response selection in retrieval-based dialogues, and our model achieves new state-of-the-art results on both datasets.