# Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning

arXiv: https://arxiv.org/abs/1705.00557 · HTML: https://mlsanity.com/paper/1705.00557/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2017.

## Built on by (most cited first, 18 of 18)

- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [RankGen: Improving Text Generation with Large Ranking Models](https://mlsanity.com/paper/2205.09726/) (2022)
- [Improving In-Context Few-Shot Learning via Self-Supervised Training](https://mlsanity.com/paper/2205.01703/) (2022)
- [SciNLI: A Corpus for Natural Language Inference on Scientific Text](https://mlsanity.com/paper/2203.06728/) (2022)
- [When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research](https://mlsanity.com/paper/2505.11855/) (2025)
- [Discourse Relation Embeddings: Representing the Relations between Discourse Segments in Social Media](https://mlsanity.com/paper/2105.01306/) (2021)
- [Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learning](https://mlsanity.com/paper/2212.08663/) (2022)
- [PAUSE: Positive and Annealed Unlabeled Sentence Embedding](https://mlsanity.com/paper/2109.03155/) (2021)
- [Fortunately, Discourse Markers Can Enhance Language Models for Sentiment Analysis](https://mlsanity.com/paper/2201.02026/) (2022)
- [Retrofitting Multilingual Sentence Embeddings with Abstract Meaning Representation](https://mlsanity.com/paper/2210.09773/) (2022)
- [AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model](https://mlsanity.com/paper/2211.11363/) (2022)
- [Distributed Marker Representation for Ambiguous Discourse Markers and Entangled Relations](https://mlsanity.com/paper/2306.10658/) (2023)
- [Leveraging Natural Supervision for Language Representation Learning and Generation](https://mlsanity.com/paper/2207.10617/) (2022)
- [Enhancing Pre-Trained Language Models with Sentence Position Embeddings for Rhetorical Roles Recognition in Legal Opinions](https://mlsanity.com/paper/2310.05276/) (2023)
- [elBERto: Self-supervised Commonsense Learning for Question Answering](https://mlsanity.com/paper/2203.09424/) (2022)
- [When Do Discourse Markers Affect Computational Sentence Understanding?](https://mlsanity.com/paper/2309.00368/) (2023)
- [DEPTH: Discourse Education through Pre-Training Hierarchically](https://mlsanity.com/paper/2405.07788/) (2024)
- [The Large Language Model GreekLegalRoBERTa](https://mlsanity.com/paper/2410.12852/) (2024)

## Abstract (arXiv)

This work presents a novel objective function for the unsupervised training of neural network sentence encoders. It exploits signals from paragraph-level discourse coherence to train these models to understand text. Our objective is purely discriminative, allowing us to train models many times faster than was possible under prior methods, and it yields models which perform well in extrinsic evaluations.