# Pretraining Methods for Dialog Context Representation Learning

arXiv: https://arxiv.org/abs/1906.00414 · HTML: https://mlsanity.com/paper/1906.00414/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2019.

## Built on by (most cited first, 30 of 35)

- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injection](https://mlsanity.com/paper/2111.14592/) (2021)
- [Dialogue State Tracking with a Language Model using Schema-Driven Prompting](https://mlsanity.com/paper/2109.07506/) (2021)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning](https://mlsanity.com/paper/2205.12673/) (2022)
- [SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generation](https://mlsanity.com/paper/2209.06664/) (2022)
- [Multi2WOZ: A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialog](https://mlsanity.com/paper/2205.10400/) (2022)
- [ConvFiT: Conversational Fine-Tuning of Pretrained Language Models](https://mlsanity.com/paper/2109.10126/) (2021)
- [Structural Pre-training for Dialogue Comprehension](https://mlsanity.com/paper/2105.10956/) (2021)
- [Effective Sequence-to-Sequence Dialogue State Tracking](https://mlsanity.com/paper/2108.13990/) (2021)
- [Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking](https://mlsanity.com/paper/2109.13620/) (2021)
- [Code-switched inspired losses for generic spoken dialog representations](https://mlsanity.com/paper/2108.12465/) (2021)
- [Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation](https://mlsanity.com/paper/2105.14445/) (2021)
- [Representation Learning for Conversational Data using Discourse Mutual Information Maximization](https://mlsanity.com/paper/2112.05787/) (2021)
- [OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue](https://mlsanity.com/paper/2209.04595/) (2022)
- [Pre-training Multi-party Dialogue Models with Latent Discourse Inference](https://mlsanity.com/paper/2305.15175/) (2023)
- [Domain Adaptation in Intent Classification Systems: A Review](https://mlsanity.com/paper/2404.14415/) (2024)
- [DialAug: Mixing up Dialogue Contexts in Contrastive Learning for Robust Conversational Modeling](https://mlsanity.com/paper/2204.07679/) (2022)
- [KPT: Keyword-guided Pre-training for Grounded Dialog Generation](https://mlsanity.com/paper/2212.01739/) (2022)
- [Bengali Intent Classification with Generative Adversarial BERT](https://mlsanity.com/paper/2312.10679/) (2023)
- [When is it permissible for artificial intelligence to lie? A trust-based approach](https://mlsanity.com/paper/2103.05434/) (2021)
- [Matching-based Term Semantics Pre-training for Spoken Patient Query Understanding](https://mlsanity.com/paper/2303.01341/) (2023)
- [Learning Locality and Isotropy in Dialogue Modeling](https://mlsanity.com/paper/2205.14583/) (2022)
- [An Empirical Study of Multitask Learning to Improve Open Domain Dialogue Systems](https://mlsanity.com/paper/2304.08115/) (2023)
- [Dialogue Agents 101: A Beginner's Guide to Critical Ingredients for Designing Effective Conversational Systems](https://mlsanity.com/paper/2307.07255/) (2023)
- [Neural Conversation Models and How to Rein Them in: A Survey of Failures and Fixes](https://mlsanity.com/paper/2308.06095/) (2023)
- [SS-MPC: A Sequence-Structured Multi-Party Conversation System](https://mlsanity.com/paper/2502.16920/) (2025)
- [Beyond Context: Large Language Models' Failure to Grasp Users' Intent](https://mlsanity.com/paper/2512.21110/) (2025)
- [A survey of neural models for the automatic analysis of conversation: Towards a better integration of the social sciences](https://mlsanity.com/paper/2203.16891/) (2022)
- [Zero-shot Cross-lingual Conversational Semantic Role Labeling](https://mlsanity.com/paper/2204.04914/) (2022)

## Abstract (arXiv)

This paper examines various unsupervised pretraining objectives for learning dialog context representations. Two novel methods of pretraining dialog context encoders are proposed, and a total of four methods are examined. Each pretraining objective is fine-tuned and evaluated on a set of downstream dialog tasks using the MultiWoz dataset and strong performance improvement is observed. Further evaluation shows that our pretraining objectives result in not only better performance, but also better convergence, models that are less data hungry and have better domain generalizability.