mlsanityData to 14 Oct 2026

TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning

arXiv 2104.06979 ↗First posted 14 Apr 2021cs.CLcited by 116

Not on the map. Why?

Abstract (arXiv)

Learning sentence embeddings often requires a large amount of labeled data. However, for most tasks and domains, labeled data is seldom available and creating it is expensive. In this work, we present a new state-of-the-art unsupervised method based on pre-trained Transformers and Sequential Denoising Auto-Encoder (TSDAE) which outperforms previous approaches by up to 6.4 points. It can achieve up to 93.1% of the performance of in-domain supervised approaches. Further, we show that TSDAE is a strong domain adaptation and pre-training method for sentence embeddings, significantly outperforming other approaches like Masked Language Model. A crucial shortcoming of previous studies is the narrow evaluation: Most work mainly evaluates on the single task of Semantic Textual Similarity (STS), which does not require any domain knowledge. It is unclear if these proposed methods generalize to other domains and tasks. We fill this gap and evaluate TSDAE and other recent approaches on four different datasets from heterogeneous domains.

Key facts. Cited by 116 papers in the map (data through 14 Oct 2026). Builds on 23 of its 46 references that are in the map. First posted on arXiv 14 Apr 2021.

Builds on

23 of 46 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  10. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  11. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  12. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
11 more
  1. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 430
  2. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
  3. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 229
  4. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 202
  5. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 193
  6. 2017Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Featurescited by 104
  7. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 91
  8. 2020On the Sentence Embeddings from Pre-trained Language Modelscited by 83
  9. 2017A Continuously Growing Dataset of Sentential Paraphrasescited by 72
  10. 2015Semi-supervised Question Retrieval with Gated Convolutionscited by 28
  11. 2018Learning Universal Sentence Representations with Mean-Max Attention Autoencodercited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 27; 2023: 26; 2024: 23; 2025: 22; 2026: 12.

116 papers.

Sort:

By area:

  1. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  2. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  3. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  4. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  5. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  6. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  7. 2022From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effectivecited by 98
  8. 2024GuardT2I: Defending Text-to-Image Models from Adversarial Promptscited by 62
  9. 2022Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformercited by 45
  10. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  11. 2022PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddingscited by 29
  12. 2023RankCSE: Unsupervised Sentence Representations Learning via Learning to Rankcited by 23
38 more
  1. 2021Virtual Augmentation Supported Contrastive Learning of Sentence Representationscited by 21
  2. 2022Just Rank: Rethinking Evaluation with Word and Sentence Similaritiescited by 20
  3. 2022COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learningcited by 20
  4. 2024The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model designcited by 20
  5. 2022Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Frameworkcited by 19
  6. 2023Nugget: Neural Agglomerative Embeddings of Textcited by 19
  7. 2024PL-MTEB: Polish Massive Text Embedding Benchmarkcited by 19
  8. 2023Tsetlin Machine Embedding: Representing Words Using Logical Expressionscited by 15
  9. 2025Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learningcited by 15
  10. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
  11. 2024Deep Learning-based Computational Job Market Analysis: A Survey on Skill Extraction and Classification from Job Postingscited by 14
  12. 2023RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Modelscited by 13
  13. 2024German Text Embedding Clustering Benchmarkcited by 13
  14. 2022Sentence Representation Learning with Generative Objective rather than Contrastive Objectivecited by 12
  15. 2023A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyondcited by 12
  16. 2023Time Series Anomaly Detection using Diffusion-based Modelscited by 12
  17. 2022Training Effective Neural Sentence Encoders from Automatically Mined Paraphrasescited by 11
  18. 2022Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddingscited by 10
  19. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  20. 2024textttMixGR: Enhancing Retriever Generalization for Scientific Domain through Complementary Granularitycited by 9
  21. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLMcited by 8
  22. 2023Learning Human-like Representations to Enable Learning Human Valuescited by 7
  23. 2024SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languagescited by 7
  24. 2022Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embeddingcited by 6
  25. 2023Learning Disentangled Semantic Spaces of Explanations via Invertible Neural Networkscited by 6
  26. 2023Length is a Curse and a Blessing for Document-level Semanticscited by 6
  27. 2024MTEB-French: Resources for French Sentence Embedding Evaluation and Analysiscited by 6
  28. 2026CSRv2: Unlocking Ultra-Sparse Embeddingscited by 6
  29. 2022AugCSE: Contrastive Sentence Embedding with Diverse Augmentationscited by 5
  30. 2022Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negativescited by 5
  31. 2023Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operationscited by 5
  32. 2023Generative User-Experience Research for Developing Domain-specific Natural Language Processing Applicationscited by 5
  33. 2023AHAM: Adapt, Help, Ask, Model -- Harvesting LLMs for literature miningcited by 5
  34. 2025CareerBERT: Matching Resumes to ESCO Jobs in a Shared Embedding Space for Generic Job Recommendationscited by 5
  35. 2025VN-MTEB: Vietnamese Massive Text Embedding Benchmarkcited by 5
  36. 2022Establishing Strong Baselines for TripClick Health Retrievalcited by 4
  37. 2022SynWMD: Syntax-aware Word Mover's Distance for Sentence Similarity Evaluationcited by 4
  38. 2022Relational Sentence Embedding for Flexible Semantic Matchingcited by 4

50 most cited of 116 All 116 (JSON)