# Mining Discourse Markers for Unsupervised Sentence Representation Learning

arXiv: https://arxiv.org/abs/1903.11850 · HTML: https://mlsanity.com/paper/1903.11850/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2019.

## Built on by (most cited first, 30 of 34)

- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models](https://mlsanity.com/paper/2203.06904/) (2022)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [In-Context Learning with Long-Context Models: An In-Depth Exploration](https://mlsanity.com/paper/2405.00200/) (2024)
- [BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation](https://mlsanity.com/paper/2103.11878/) (2021)
- [Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metric](https://mlsanity.com/paper/2209.13654/) (2022)
- [Pre-Training to Learn in Context](https://mlsanity.com/paper/2305.09137/) (2023)
- [ChatGPT Evaluation on Sentence Level Relations: A Focus on Temporal, Causal, and Discourse Relations](https://mlsanity.com/paper/2304.14827/) (2023)
- [Discourse Probing of Pretrained Language Models](https://mlsanity.com/paper/2104.05882/) (2021)
- [Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning](https://mlsanity.com/paper/2104.08808/) (2021)
- [Exploring Universal Intrinsic Task Subspace via Prompt Tuning](https://mlsanity.com/paper/2110.07867/) (2021)
- [Downstream Datasets Make Surprisingly Good Pretraining Corpora](https://mlsanity.com/paper/2209.14389/) (2022)
- [Pragmatic competence of pre-trained language models through the lens of discourse connectives](https://mlsanity.com/paper/2109.12951/) (2021)
- [IGA : An Intent-Guided Authoring Assistant](https://mlsanity.com/paper/2104.07000/) (2021)
- [Eliciting and Understanding Cross-Task Skills with Task-Level Mixture-of-Experts](https://mlsanity.com/paper/2205.12701/) (2022)
- [Annotation-Inspired Implicit Discourse Relation Classification with Auxiliary Discourse Connective Generation](https://mlsanity.com/paper/2306.06480/) (2023)
- [Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?](https://mlsanity.com/paper/2302.08143/) (2023)
- [Top-down Discourse Parsing via Sequence Labelling](https://mlsanity.com/paper/2102.02080/) (2021)
- [Discourse Relation Embeddings: Representing the Relations between Discourse Segments in Social Media](https://mlsanity.com/paper/2105.01306/) (2021)
- [AMERICANO: Argument Generation with Discourse-driven Decomposition and Agent Interaction](https://mlsanity.com/paper/2310.20352/) (2023)
- [Few-shot Adaptation Works with UnpredicTable Data](https://mlsanity.com/paper/2208.01009/) (2022)
- [Fortunately, Discourse Markers Can Enhance Language Models for Sentiment Analysis](https://mlsanity.com/paper/2201.02026/) (2022)
- [Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learning](https://mlsanity.com/paper/2312.17055/) (2023)
- [Deep learning for sentence clustering in essay grading support](https://mlsanity.com/paper/2104.11556/) (2021)
- [Analysis and Prediction of NLP Models Via Task Embeddings](https://mlsanity.com/paper/2112.05647/) (2021)
- [Distributed Marker Representation for Ambiguous Discourse Markers and Entangled Relations](https://mlsanity.com/paper/2306.10658/) (2023)
- [Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systems](https://mlsanity.com/paper/2602.23266/) (2026)
- [KnowDA: All-in-One Knowledge Mixture Model for Data Augmentation in Low-Resource NLP](https://mlsanity.com/paper/2206.10265/) (2022)
- [DiscoSense: Commonsense Reasoning with Discourse Connectives](https://mlsanity.com/paper/2210.12478/) (2022)

## Abstract (arXiv)

Current state of the art systems in NLP heavily rely on manually annotated datasets, which are expensive to construct. Very little work adequately exploits unannotated data -- such as discourse markers between sentences -- mainly because of data sparseness and ineffective extraction methods. In the present work, we propose a method to automatically discover sentence pairs with relevant discourse markers, and apply it to massive amounts of data. Our resulting dataset contains 174 discourse markers with at least 10k examples each, even for rare markers such as coincidentally or amazingly We use the resulting data as supervision for learning transferable sentence embeddings. In addition, we show that even though sentence representation learning through prediction of discourse markers yields state of the art results across different transfer tasks, it is not clear that our models made use of the semantic relation between sentences, thus leaving room for further improvements. Our datasets are publicly available (https://github.com/synapse-developpement/Discovery)