# PAUSE: Positive and Annealed Unlabeled Sentence Embedding

arXiv: https://arxiv.org/abs/2109.03155 · HTML: https://mlsanity.com/paper/2109.03155/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 59 references that are in the map.
- First posted on arXiv 7 Sep 2021.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [Positive-Unlabeled Learning with Non-Negative Risk Estimator](https://mlsanity.com/paper/1703.00593/) (2017)
- [An efficient framework for learning sentence representations](https://mlsanity.com/paper/1803.02893/) (2018)
- [Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features](https://mlsanity.com/paper/1703.02507/) (2017)
- [Self-Guided Contrastive Learning for BERT Sentence Representations](https://mlsanity.com/paper/2106.07345/) (2021)
- [Learning Distributed Representations of Sentences from Unlabelled Data](https://mlsanity.com/paper/1602.03483/) (2016)
- [An Unsupervised Sentence Embedding Method by Mutual Information Maximization](https://mlsanity.com/paper/2009.12061/) (2020)
- [Self-PU: Self Boosted and Calibrated Positive-Unlabeled Training](https://mlsanity.com/paper/2006.11280/) (2020)
- [Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model](https://mlsanity.com/paper/1810.12836/) (2018)
- [Classification from Positive, Unlabeled and Biased Negative Data](https://mlsanity.com/paper/1810.00846/) (2018)
- [Word Mover's Embedding: From Word2Vec to Document Embedding](https://mlsanity.com/paper/1811.01713/) (2018)
- [Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning](https://mlsanity.com/paper/1705.00557/) (2017)
- [Efficient Vector Representation for Documents through Corruption](https://mlsanity.com/paper/1707.02377/) (2017)
- [Better Word Embeddings by Disentangling Contextual n-Gram Information](https://mlsanity.com/paper/1904.05033/) (2019)
- [Parameter-free Sentence Embedding via Orthogonal Basis](https://mlsanity.com/paper/1810.00438/) (2018)
- [LearningWord Embeddings for Low-resource Languages by PU Learning](https://mlsanity.com/paper/1805.03366/) (2018)

## Built on by (most cited first, 5 of 5)

- [Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2209.09433/) (2022)
- [A Unified Positive-Unlabeled Learning Framework for Document-Level Relation Extraction with Different Levels of Labeling](https://mlsanity.com/paper/2210.08709/) (2022)
- [Extracting Sentence Embeddings from Pretrained Transformer Models](https://mlsanity.com/paper/2408.08073/) (2024)
- [CompanyKG: A Large-Scale Heterogeneous Graph for Company Similarity Quantification](https://mlsanity.com/paper/2306.10649/) (2023)
- [A Positive-Unlabeled Metric Learning Framework for Document-Level Relation Extraction with Incomplete Labeling](https://mlsanity.com/paper/2306.14806/) (2023)

## Abstract (arXiv)

Sentence embedding refers to a set of effective and versatile techniques for converting raw text into numerical vector representations that can be used in a wide range of natural language processing (NLP) applications. The majority of these techniques are either supervised or unsupervised. Compared to the unsupervised methods, the supervised ones make less assumptions about optimization objectives and usually achieve better results. However, the training requires a large amount of labeled sentence pairs, which is not available in many industrial scenarios. To that end, we propose a generic and end-to-end approach -- PAUSE (Positive and Annealed Unlabeled Sentence Embedding), capable of learning high-quality sentence embeddings from a partially labeled dataset. We experimentally show that PAUSE achieves, and sometimes surpasses, state-of-the-art results using only a small fraction of labeled sentence pairs on various benchmark tasks. When applied to a real industrial use case where labeled samples are scarce, PAUSE encourages us to extend our dataset without the liability of extensive manual annotation work.