mlsanityData to 14 Oct 2026

Supervised Graph Contrastive Pretraining for Text Classification

arXiv 2112.11389 ↗First posted 21 Dec 2021cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Contrastive pretraining techniques for text classification has been largely studied in an unsupervised setting. However, oftentimes labeled data from related tasks which share label semantics with current task is available. We hypothesize that using this labeled data effectively can lead to better generalization on current task. In this paper, we propose a novel way to effectively utilize labeled data from related tasks with a graph based supervised contrastive learning approach. We formulate a token-graph by extrapolating the supervised information from examples to tokens. Our formulation results in an embedding space where tokens with high/low probability of belonging to same class are near/further-away from one another. We also develop detailed theoretical insights which serve as a motivation for our method. In our experiments with 13 datasets, we show our method outperforms pretraining schemes by 2.5% and also example-level contrastive learning based formulation by 1.8% on average. In addition, we show cross-domain effectiveness of our method in a zero-shot setting by 3.91% on average. Lastly, we also demonstrate our method can be used as a noisy teacher in a knowledge distillation setting to significantly improve performance of transformer based models in low labeled data regime by 4.57% on average.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 28 references that are in the map. First posted on arXiv 21 Dec 2021.

Builds on

17 of 28 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  5. 2020Supervised Contrastive Learningcited by 2,829
  6. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
  7. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  8. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  9. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 276
  10. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
  11. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  12. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
5 more
  1. 2018Domain Adaptation with Adversarial Training and Graph Embeddingscited by 30
  2. 2021Token-Level Supervised Contrastive Learning for Punctuation Restorationcited by 14
  3. 2020A Simple and Effective Self-Supervised Contrastive Learning Framework for Aspect Detectioncited by 13
  4. 2020Cross-Domain Sentiment Classification with Contrastive Learning and Mutual Information Maximizationcited by 9
  5. 2021Sentence Embeddings using Supervised Contrastive Learningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Distilling Large Language Models for Text-Attributed Graph Learning
  2. 2023Improving Contrastive Learning of Sentence Embeddings from AI Feedback
  3. 2025The Efficiency of Pre-training with Objective Masking in Pseudo Labeling for Semi-Supervised Text Classification
  4. 2018Mixture of Expert/Imitator Networks: Scalable Semi-supervised Learning Framework
  5. 2025Contrast-CAT: Contrasting Activations for Enhanced Interpretability in Transformer-based Text Classifiers
  6. 2024Contrastive Learning in Distilled Models
  7. 2022Contrastive Graph Few-Shot Learning
  8. 2025Efficient Text-Attributed Graph Learning through Selective Annotation and Graph Alignment