# Dial-MAE: ConTextual Masked Auto-Encoder for Retrieval-based Dialogue Systems

arXiv: https://arxiv.org/abs/2306.04357 · HTML: https://mlsanity.com/paper/2306.04357/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 7 Jun 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems](https://mlsanity.com/paper/1506.08909/) (2015)
- [Condenser: a Pre-training Architecture for Dense Retrieval](https://mlsanity.com/paper/2104.08253/) (2021)
- [RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder](https://mlsanity.com/paper/2205.12035/) (2022)
- [Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots](https://mlsanity.com/paper/1612.01627/) (2016)
- [Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots](https://mlsanity.com/paper/2004.03588/) (2020)
- [Modeling Multi-turn Conversation with Deep Utterance Aggregation](https://mlsanity.com/paper/1806.09102/) (2018)
- [Tevatron: An Efficient and Flexible Toolkit for Dense Retrieval](https://mlsanity.com/paper/2203.05765/) (2022)
- [Do Response Selection Models Really Know What's Next? Utterance Manipulation Strategies for Multi-turn Response Selection](https://mlsanity.com/paper/2009.04703/) (2020)
- [Learning an Effective Context-Response Matching Model with Self-Supervised Tasks for Retrieval-based Dialogues](https://mlsanity.com/paper/2009.06265/) (2020)
- [ConTextual Masked Auto-Encoder for Dense Passage Retrieval](https://mlsanity.com/paper/2208.07670/) (2022)
- [Improved Deep Learning Baselines for Ubuntu Corpus Dialogs](https://mlsanity.com/paper/1510.03753/) (2015)
- [Exploring Dense Retrieval for Dialogue Response Selection](https://mlsanity.com/paper/2110.06612/) (2021)
- [Sequential Attention-based Network for Noetic End-to-End Response Selection](https://mlsanity.com/paper/1901.02609/) (2019)
- [Two-Level Supervised Contrastive Learning for Response Selection in Multi-Turn Dialogue](https://mlsanity.com/paper/2203.00793/) (2022)

## Built on by (most cited first, 2 of 2)

- [Modeling the One-to-Many Property in Open-Domain Dialogue with LLMs](https://mlsanity.com/paper/2506.15131/) (2025)
- [A Diverse and Effective Retrieval-Based Debt Collection System with Expert Knowledge](https://mlsanity.com/paper/2504.06273/) (2025)

## Abstract (arXiv)

Dialogue response selection aims to select an appropriate response from several candidates based on a given user and system utterance history. Most existing works primarily focus on post-training and fine-tuning tailored for cross-encoders. However, there are no post-training methods tailored for dense encoders in dialogue response selection. We argue that when the current language model, based on dense dialogue systems (such as BERT), is employed as a dense encoder, it separately encodes dialogue context and response, leading to a struggle to achieve the alignment of both representations. Thus, we propose Dial-MAE (Dialogue Contextual Masking Auto-Encoder), a straightforward yet effective post-training technique tailored for dense encoders in dialogue response selection. Dial-MAE uses an asymmetric encoder-decoder architecture to compress the dialogue semantics into dense vectors, which achieves better alignment between the features of the dialogue context and response. Our experiments have demonstrated that Dial-MAE is highly effective, achieving state-of-the-art performance on two commonly evaluated benchmarks.