# ESCL: Equivariant Self-Contrastive Learning for Sentence Representations

arXiv: https://arxiv.org/abs/2303.05143 · HTML: https://mlsanity.com/paper/2303.05143/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 25 references that are in the map.
- First posted on arXiv 9 Mar 2023.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings](https://mlsanity.com/paper/2204.10298/) (2022)
- [Self-Guided Contrastive Learning for BERT Sentence Representations](https://mlsanity.com/paper/2106.07345/) (2021)
- [On the Sentence Embeddings from Pre-trained Language Models](https://mlsanity.com/paper/2011.05864/) (2020)
- [Equivariant Contrastive Learning](https://mlsanity.com/paper/2111.00899/) (2021)
- [SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samples](https://mlsanity.com/paper/2201.05979/) (2022)
- [Virtual Augmentation Supported Contrastive Learning of Sentence Representations](https://mlsanity.com/paper/2110.08552/) (2021)
- [EASE: Entity-Aware Contrastive Learning of Sentence Embedding](https://mlsanity.com/paper/2205.04260/) (2022)
- [SCD: Self-Contrastive Decorrelation for Sentence Embeddings](https://mlsanity.com/paper/2203.07847/) (2022)
- [VIRT: Improving Representation-based Models for Text Matching through Virtual Interaction](https://mlsanity.com/paper/2112.04195/) (2021)
- [MetricBERT: Text Representation Learning via Self-Supervised Triplet Training](https://mlsanity.com/paper/2208.06610/) (2022)

## Built on by (most cited first, 1 of 1)

- [LaCViT: A Label-aware Contrastive Fine-tuning Framework for Vision Transformers](https://mlsanity.com/paper/2303.18013/) (2023)

## Abstract (arXiv)

Previous contrastive learning methods for sentence representations often focus on insensitive transformations to produce positive pairs, but neglect the role of sensitive transformations that are harmful to semantic representations. Therefore, we propose an Equivariant Self-Contrastive Learning (ESCL) method to make full use of sensitive transformations, which encourages the learned representations to be sensitive to certain types of transformations with an additional equivariant learning task. Meanwhile, in order to improve practicability and generality, ESCL simplifies the implementations of traditional equivariant contrastive methods to share model parameters from the perspective of multi-task learning. We evaluate our ESCL on semantic textual similarity tasks. The proposed method achieves better results while using fewer learning parameters compared to previous methods.