# Large-Context Conversational Representation Learning: Self-Supervised Learning for Conversational Documents

arXiv: https://arxiv.org/abs/2102.08147 · HTML: https://mlsanity.com/paper/2102.08147/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 16 Feb 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues](https://mlsanity.com/paper/1605.06069/) (2016)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [Semi-supervised sequence tagging with bidirectional language models](https://mlsanity.com/paper/1705.00108/) (2017)
- [Exploiting Cross-Sentence Context for Neural Machine Translation](https://mlsanity.com/paper/1704.04347/) (2017)
- [HiGRU: Hierarchical Gated Recurrent Units for Utterance-level Emotion Recognition](https://mlsanity.com/paper/1904.04446/) (2019)
- [Document Context Neural Machine Translation with Memory Networks](https://mlsanity.com/paper/1711.03688/) (2017)
- [Does Neural Machine Translation Benefit from Larger Context?](https://mlsanity.com/paper/1704.05135/) (2017)
- [Dialog-context aware end-to-end speech recognition](https://mlsanity.com/paper/1808.02171/) (2018)
- [Dialogue Act Classification with Context-Aware Self-Attention](https://mlsanity.com/paper/1904.02594/) (2019)
- [Dialogue Act Recognition via CRF-Attentive Structured Network](https://mlsanity.com/paper/1711.05568/) (2017)
- [Dialogue Act Sequence Labeling using Hierarchical encoder with CRF](https://mlsanity.com/paper/1709.04250/) (2017)
- [Larger-Context Language Modelling](https://mlsanity.com/paper/1511.03729/) (2015)
- [Modeling Long-Range Context for Concurrent Dialogue Acts Recognition](https://mlsanity.com/paper/1909.00521/) (2019)

## Built on by (most cited first, 1 of 1)

- [Hierarchical Knowledge Distillation for Dialogue Sequence Labeling](https://mlsanity.com/paper/2111.10957/) (2021)

## Abstract (arXiv)

This paper presents a novel self-supervised learning method for handling conversational documents consisting of transcribed text of human-to-human conversations. One of the key technologies for understanding conversational documents is utterance-level sequential labeling, where labels are estimated from the documents in an utterance-by-utterance manner. The main issue with utterance-level sequential labeling is the difficulty of collecting labeled conversational documents, as manual annotations are very costly. To deal with this issue, we propose large-context conversational representation learning (LC-CRL), a self-supervised learning method specialized for conversational documents. A self-supervised learning task in LC-CRL involves the estimation of an utterance using all the surrounding utterances based on large-context language modeling. In this way, LC-CRL enables us to effectively utilize unlabeled conversational documents and thereby enhances the utterance-level sequential labeling. The results of experiments on scene segmentation tasks using contact center conversational datasets demonstrate the effectiveness of the proposed method.