# Pair-Level Supervised Contrastive Learning for Natural Language Inference

arXiv: https://arxiv.org/abs/2201.10927 · HTML: https://mlsanity.com/paper/2201.10927/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 26 Jan 2022.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [Towards Natural Language Interfaces for Data Visualization: A Survey](https://mlsanity.com/paper/2109.03506/) (2021)
- [SelfORE: Self-supervised Relational Feature Learning for Open Relation Extraction](https://mlsanity.com/paper/2004.02438/) (2020)
- [Neural Natural Language Inference Models Enhanced with External Knowledge](https://mlsanity.com/paper/1711.04289/) (2017)
- [Semi-supervised Relation Extraction via Incremental Meta Self-Training](https://mlsanity.com/paper/2010.16410/) (2020)
- [Natural Language Inference over Interaction Space](https://mlsanity.com/paper/1709.04348/) (2017)
- [Gradient Imitation Reinforcement Learning for Low Resource Relation Extraction](https://mlsanity.com/paper/2109.06415/) (2021)

## Built on by (most cited first, 12 of 12)

- [HiURE: Hierarchical Exemplar Contrastive Learning for Unsupervised Relation Extraction](https://mlsanity.com/paper/2205.02225/) (2022)
- [Semantic Enhanced Text-to-SQL Parsing via Iteratively Learning Schema Linking Graph](https://mlsanity.com/paper/2208.03903/) (2022)
- [PART: Pre-trained Authorship Representation Transformer](https://mlsanity.com/paper/2209.15373/) (2022)
- [Understanding writing style in social media with a supervised contrastively pre-trained transformer](https://mlsanity.com/paper/2310.11081/) (2023)
- [GDA: Generative Data Augmentation Techniques for Relation Extraction Tasks](https://mlsanity.com/paper/2305.16663/) (2023)
- [Towards Discriminative Representations with Contrastive Instances for Real-Time UAV Tracking](https://mlsanity.com/paper/2308.11450/) (2023)
- [A Multi-level Supervised Contrastive Learning Framework for Low-Resource Natural Language Inference](https://mlsanity.com/paper/2205.15550/) (2022)
- [Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizer](https://mlsanity.com/paper/2305.12761/) (2023)
- [Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis](https://mlsanity.com/paper/2305.16166/) (2023)
- [Entity-to-Text based Data Augmentation for various Named Entity Recognition Tasks](https://mlsanity.com/paper/2210.10343/) (2022)
- [Give Me More Details: Improving Fact-Checking with Latent Retrieval](https://mlsanity.com/paper/2305.16128/) (2023)
- [Scene Graph Modification as Incremental Structure Expanding](https://mlsanity.com/paper/2209.09093/) (2022)

## Abstract (arXiv)

Natural language inference (NLI) is an increasingly important task for natural language understanding, which requires one to infer the relationship between the sentence pair (premise and hypothesis). Many recent works have used contrastive learning by incorporating the relationship of the sentence pair from NLI datasets to learn sentence representation. However, these methods only focus on comparisons with sentence-level representations. In this paper, we propose a Pair-level Supervised Contrastive Learning approach (PairSCL). We adopt a cross attention module to learn the joint representations of the sentence pairs. A contrastive learning objective is designed to distinguish the varied classes of sentence pairs by pulling those in one class together and pushing apart the pairs in other classes. We evaluate PairSCL on two public datasets of NLI where the accuracy of PairSCL outperforms other methods by 2.1% on average. Furthermore, our method outperforms the previous state-of-the-art method on seven transfer tasks of text classification.