# Cross-Thought for Sentence Encoder Pre-training

arXiv: https://arxiv.org/abs/2010.03652 · HTML: https://mlsanity.com/paper/2010.03652/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2020.

## Built on by (most cited first, 8 of 8)

- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [Trial2Vec: Zero-Shot Clinical Trial Document Similarity Search using Self-Supervision](https://mlsanity.com/paper/2206.14719/) (2022)
- [Sentence Representation Learning with Generative Objective rather than Contrastive Objective](https://mlsanity.com/paper/2210.08474/) (2022)
- [REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training](https://mlsanity.com/paper/2105.04201/) (2021)
- [CoLLEGe: Concept Embedding Generation for Large Language Models](https://mlsanity.com/paper/2403.15362/) (2024)
- [Enhancing Natural Language Representation with Large-Scale Out-of-Domain Commonsense](https://mlsanity.com/paper/2109.02572/) (2021)
- [SECRET: Semi-supervised Clinical Trial Document Similarity Search](https://mlsanity.com/paper/2505.10780/) (2025)
- [Generative or Contrastive? Phrase Reconstruction for Better Sentence Representation Learning](https://mlsanity.com/paper/2204.09358/) (2022)

## Abstract (arXiv)

In this paper, we propose Cross-Thought, a novel approach to pre-training sequence encoder, which is instrumental in building reusable sequence embeddings for large-scale NLP tasks such as question answering. Instead of using the original signals of full sentences, we train a Transformer-based sequence encoder over a large set of short sequences, which allows the model to automatically select the most useful information for predicting masked words. Experiments on question answering and textual entailment tasks demonstrate that our pre-trained encoder can outperform state-of-the-art encoders trained with continuous sentence signals as well as traditional masked language modeling baselines. Our proposed approach also achieves new state of the art on HotpotQA (full-wiki setting) by improving intermediate information retrieval performance.