mlsanityData to 7 Oct 2026

Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models

arXiv 2108.08877 ↗First posted 19 Aug 2021cs.CLcited by 462

Not on the map. Why?

Cited by 23 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We provide the first exploration of sentence embeddings from text-to-text transformers (T5). Sentence embeddings are broadly useful for language processing tasks. While T5 achieves impressive performance on language tasks cast as sequence-to-sequence mapping problems, it is unclear how to produce sentence embeddings from encoder-decoder models. We investigate three methods for extracting T5 sentence embeddings: two utilize only the T5 encoder and one uses the full T5 encoder-decoder model. To support our investigation, we establish a new sentence representation transfer benchmark, SentGLUE, which extends the SentEval toolkit to nine tasks from the GLUE benchmark. Our encoder-only models outperforms Sentence-BERT and SimCSE sentence embeddings on both SentEval and SentGLUE transfer tasks, including semantic textual similarity (STS). Scaling up T5 from millions to billions of parameters is found to produce consistent further improvements. Finally, our encoder-decoder method achieves a new state-of-the-art on STS when using sentence embeddings. Our models are released at https://tfhub.dev/google/collections/sentence-t5/1.

Key facts. Cited by 462 papers in the map (data through 7 Oct 2026). Builds on 30 of its 39 references that are in the map. First posted on arXiv 19 Aug 2021.

Builds on

30 of 39 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
18 more
  1. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  2. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  3. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  4. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  5. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  6. 2018Universal Sentence Encodercited by 485
  7. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  8. 2015Skip-Thought Vectorscited by 321
  9. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  10. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 228
  11. 2018An efficient framework for learning sentence representationscited by 197
  12. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  13. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  14. 2015Towards Universal Paraphrastic Sentence Embeddingscited by 59
  15. 2018End-to-End Retrieval in Continuous Spacecited by 46
  16. 2018Learning Semantic Textual Similarity from Conversationscited by 41
  17. 2020Universal Sentence Representation Learning with Conditional Masked Language Modelcited by 34
  18. 2019ReQA: An Evaluation for End-to-End Answer Retrieval Modelscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 32; 2023: 105; 2024: 144; 2025: 106; 2026: 69.

462 papers.

Sort:

By area:

  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  2. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  3. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  4. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  5. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  6. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  7. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  8. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  9. 2023Improving Text Embeddings with Large Language Modelscited by 295
  10. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  11. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  12. 2024Generative Representational Instruction Tuningcited by 227
38 more
  1. 2023Recommender Systems with Generative Retrievalcited by 191
  2. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  3. 2023Denoising Diffusion Autoencoders are Unified Self-supervised Learnerscited by 128
  4. 2023Large Language Models in Law: A Surveycited by 128
  5. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 123
  6. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 114
  7. 2025Gemini Embedding: Generalizable Embeddings from Geminicited by 93
  8. 2024ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehiclescited by 92
  9. 2024How to Train Data-Efficient LLMscited by 91
  10. 2022SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Modelscited by 90
  11. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  12. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  13. 2023Scaling Sentence Embeddings with Large Language Modelscited by 83
  14. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  15. 2023Sentence Embedding Leaks More Information than You Expect: Generative Embedding Inversion Attack to Recover the Whole Sentencecited by 81
  16. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 81
  17. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  18. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  19. 2023ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendationcited by 76
  20. 2024Repetition Improves Language Model Embeddingscited by 70
  21. 2023ClusterLLM: Large Language Models as a Guide for Text Clusteringcited by 69
  22. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  23. 2021CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learningcited by 68
  24. 2024LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wildcited by 68
  25. 2023Harnessing large-language models to generate private synthetic textcited by 65
  26. 2023RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Modelscited by 62
  27. 2022Dialog Inpainting: Turning Documents into Dialogscited by 56
  28. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  29. 2023Better Zero-Shot Reasoning with Self-Adaptive Promptingcited by 52
  30. 2023Teach LLMs to Personalize -- An Approach inspired by Writing Educationcited by 48
  31. 2022Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusterscited by 44
  32. 2023Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Languagecited by 44
  33. 2023Large Language Models can be Guided to Evade AI-Generated Text Detectioncited by 43
  34. 2024PoCo: Policy Composition from and for Heterogeneous Robot Learningcited by 42
  35. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  36. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 38
  37. 2023LMDX: Language Model-based Document Information Extraction and Localizationcited by 37
  38. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37

50 most cited of 462 All 462 (JSON)