mlsanityData to 14 Oct 2026

Text2Token: Unsupervised Text Representation Learning with Token Target Prediction

arXiv 2510.10224 ↗First posted 11 Oct 2025cs.CL cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

Unsupervised text representation learning (TRL) is a fundamental task in natural language processing, which is beneficial for improving search and recommendations with the web's unlabeled texts. A recent empirical study finds that the high-quality representation aligns with the key token of the input text, uncovering the potential connection between representation space and vocabulary space. Inspired by the findings, we revisit the generative tasks and develop an unsupervised generative framework for TRL, Text2Token. The framework is based on the token target prediction task, utilizing carefully constructed target token distribution as supervisory signals. To construct the high-quality target token distribution, we analyze the token-alignment properties with advanced embedders and identify two essential categories of key tokens: (1) the meaningful tokens in the text and (2) semantically derived tokens beyond the text. Based on these insights, we propose two methods -- data-driven and model-derived -- to construct synthetic token targets from data or the LLM backbone. Experiments on the MTEB v2 benchmark demonstrate that Text2Token achieves performance competitive with the state-of-the-art embedder with unsupervised contrastive learning, LLM2Vec. Our analysis further shows that vocabulary and representation spaces optimize together and toward the optimum solution during training, providing new ideas and insights for future work.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 11 Oct 2025.

Builds on

32 of 42 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  7. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  8. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  9. 2015Skip-Thought Vectorscited by 322
  10. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  11. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 132
  12. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
20 more
  1. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  2. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 97
  3. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 91
  4. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  5. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 83
  6. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 78
  7. 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 74
  8. 2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
  9. 2020PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrievalcited by 40
  10. 2023BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddingscited by 32
  11. 2022PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddingscited by 29
  12. 2021B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrievalcited by 25
  13. 2022Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embeddingcited by 25
  14. 2023RankCSE: Unsupervised Sentence Representations Learning via Learning to Rankcited by 23
  15. 2023Alleviating Over-smoothing for Unsupervised Sentence Representationcited by 16
  16. 2021Smoothed Contrastive Learning for Unsupervised Sentence Embeddingcited by 15
  17. 2022Sentence Representation Learning with Generative Objective rather than Contrastive Objectivecited by 12
  18. 2024A Text is Worth Several Tokens: Text Embedding from LLMs Secretly Aligns Well with The Key Tokenscited by 8
  19. 2022An Information Minimization Based Contrastive Learning Model for Unsupervised Sentence Embeddings Learningcited by 6
  20. 2023Clustering-Aware Negative Sampling for Unsupervised Sentence Representationcited by 5

Built on by

Not cited yet.