# Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

arXiv: https://arxiv.org/abs/1804.00079 · HTML: https://mlsanity.com/paper/1804.00079/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2018.

## Built on by (most cited first, 30 of 52)

- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning](https://mlsanity.com/paper/2206.09363/) (2022)
- [DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training](https://mlsanity.com/paper/2303.03032/) (2023)
- [Multi-Task Learning in Natural Language Processing: An Overview](https://mlsanity.com/paper/2109.09138/) (2021)
- [Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articles](https://mlsanity.com/paper/2103.05121/) (2021)
- [PaCo: Parameter-Compositional Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2210.11653/) (2022)
- [Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning](https://mlsanity.com/paper/2205.01376/) (2022)
- [C2-CRS: Coarse-to-Fine Contrastive Learning for Conversational Recommender System](https://mlsanity.com/paper/2201.02732/) (2022)
- [Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models](https://mlsanity.com/paper/2104.05115/) (2021)
- [Paraphrastic Representations at Scale](https://mlsanity.com/paper/2104.15114/) (2021)
- [Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)](https://mlsanity.com/paper/2307.10246/) (2023)
- [Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings](https://mlsanity.com/paper/2402.17016/) (2024)
- [Pareto Set Learning for Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2501.06773/) (2025)
- [Identifying Machine-Paraphrased Plagiarism](https://mlsanity.com/paper/2103.11909/) (2021)
- [MVP: Multi-task Supervised Pre-training for Natural Language Generation](https://mlsanity.com/paper/2206.12131/) (2022)
- [Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2201.00768/) (2022)
- [FacTeR-Check: Semi-automated fact-checking through Semantic Similarity and Natural Language Inference](https://mlsanity.com/paper/2110.14532/) (2021)
- [Multi-Task Learning of Object State Changes from Uncurated Videos](https://mlsanity.com/paper/2211.13500/) (2022)
- [Improving Conversational Recommendation Systems via Counterfactual Data Simulation](https://mlsanity.com/paper/2306.02842/) (2023)
- [Pre-training Universal Language Representation](https://mlsanity.com/paper/2105.14478/) (2021)
- [Evaluating Metrics for Bias in Word Embeddings](https://mlsanity.com/paper/2111.07864/) (2021)
- [UFO: Unified Feature Optimization](https://mlsanity.com/paper/2207.10341/) (2022)
- [Best-Effort Adaptation](https://mlsanity.com/paper/2305.05816/) (2023)
- [Learning to Recognize Actions on Objects in Egocentric Video with Attention Dictionaries](https://mlsanity.com/paper/2102.08065/) (2021)
- [GDOD: Effective Gradient Descent using Orthogonal Decomposition for Multi-Task Learning](https://mlsanity.com/paper/2301.13465/) (2023)
- [Majority Voting with Bidirectional Pre-translation For Bitext Retrieval](https://mlsanity.com/paper/2103.06369/) (2021)
- [SILT: Efficient transformer training for inter-lingual inference](https://mlsanity.com/paper/2103.09635/) (2021)
- [Alleviating the Long-Tail Problem in Conversational Recommender Systems](https://mlsanity.com/paper/2307.11650/) (2023)
- [The Daunting Dilemma with Sentence Encoders: Success on Standard Benchmarks, Failure in Capturing Basic Semantic Properties](https://mlsanity.com/paper/2309.03747/) (2023)
- [Deep Learning-based Sentiment Classification: A Comparative Survey](https://mlsanity.com/paper/2312.17253/) (2023)

## Abstract (arXiv)

A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner. These representations are typically used as general purpose features for words across a range of NLP problems. However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem. Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations. In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model. We train this model on several data sources with multiple training objectives on over 100 million sentences. Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods. We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations.