# Unsupervised Contextualized Document Representation

arXiv: https://arxiv.org/abs/2109.10509 · HTML: https://mlsanity.com/paper/2109.10509/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 34 references that are in the map.
- First posted on arXiv 22 Sep 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [Balancing Objectives in Counseling Conversations: Advancing Forwards or Looking Backwards](https://mlsanity.com/paper/2005.04245/) (2020)
- [Breaking Sticks and Ambiguities with Adaptive Skip-gram](https://mlsanity.com/paper/1502.07257/) (2015)
- [Product Classification in E-Commerce using Distributional Semantics](https://mlsanity.com/paper/1606.06083/) (2016)
- [Document Similarity for Texts of Varying Lengths via Hidden Topics](https://mlsanity.com/paper/1903.10675/) (2019)
- [Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings](https://mlsanity.com/paper/1705.00364/) (2017)
- [P-SIF: Document Embeddings Using Partition Averaging](https://mlsanity.com/paper/2005.09069/) (2020)
- [Improving Document Classification with Multi-Sense Embeddings](https://mlsanity.com/paper/1911.07918/) (2019)
- [Words are not Equal: Graded Weighting Model for building Composite Document Vectors](https://mlsanity.com/paper/1512.03549/) (2015)

## Built on by (most cited first, 1 of 1)

- [Extracting Sentence Embeddings from Pretrained Transformer Models](https://mlsanity.com/paper/2408.08073/) (2024)

## Abstract (arXiv)

Several NLP tasks need the effective representation of text documents. Arora et. al., 2017 demonstrate that simple weighted averaging of word vectors frequently outperforms neural models. SCDV (Mekala et. al., 2017) further extends this from sentences to documents by employing soft and sparse clustering over pre-computed word vectors. However, both techniques ignore the polysemy and contextual character of words. In this paper, we address this issue by proposing SCDV+BERT(ctxd), a simple and effective unsupervised representation that combines contextualized BERT (Devlin et al., 2019) based word embedding for word sense disambiguation with SCDV soft clustering approach. We show that our embeddings outperform original SCDV, pre-train BERT, and several other baselines on many classification datasets. We also demonstrate our embeddings effectiveness on other tasks, such as concept matching and sentence similarity. In addition, we show that SCDV+BERT(ctxd) outperforms fine-tune BERT and different embedding approaches in scenarios with limited data and only few shots examples.