mlsanityData to 7 Oct 2026

Text Transformations in Contrastive Self-Supervised Learning: A Review

arXiv 2203.12000 ↗First posted 22 Mar 2022cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Contrastive self-supervised learning has become a prominent technique in representation learning. The main step in these methods is to contrast semantically similar and dissimilar pairs of samples. However, in the domain of Natural Language Processing (NLP), the augmentation methods used in creating similar pairs with regard to contrastive learning (CL) assumptions are challenging. This is because, even simply modifying a word in the input might change the semantic meaning of the sentence, and hence, would violate the distributional hypothesis. In this review paper, we formalize the contrastive learning framework, emphasize the considerations that need to be addressed in the data transformation step, and review the state-of-the-art methods and evaluations for contrastive representation learning in NLP. Finally, we describe some challenges and potential directions for learning better text representations using contrastive methods.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 50 of its 69 references that are in the map. First posted on arXiv 22 Mar 2022.

Builds on

50 of 69 references.

References · 50

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  5. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  6. 2020Supervised Contrastive Learningcited by 2,803
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  11. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  12. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
38 more
  1. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  2. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  3. 2020Self-supervised Learning: Generative or Contrastivecited by 693
  4. 2017Poincar'e Embeddings for Learning Hierarchical Representationscited by 674
  5. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  6. 2020Contrastive Learning with Hard Negative Samplescited by 619
  7. 2019On Variational Bounds of Mutual Informationcited by 550
  8. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  9. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  10. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  11. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  12. 2020Debiased Contrastive Learningcited by 354
  13. 2015Skip-Thought Vectorscited by 321
  14. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  15. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  16. 2020What Should Not Be Contrastive in Contrastive Learningcited by 224
  17. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  18. 2014Learning Fine-grained Image Similarity with Deep Rankingcited by 189
  19. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  20. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  21. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  22. 2019It's All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitutioncited by 102
  23. 2021The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanationscited by 86
  24. 2020Causal Interpretability for Machine Learning -- Problems, Methods and Evaluationcited by 82
  25. 2018Embedding Text in Hyperbolic Spacescited by 76
  26. 2020Learning What Makes a Difference from Counterfactual Examples and Gradient Supervisioncited by 66
  27. 2020Conditional Negative Sampling for Contrastive Learning of Visual Representationscited by 57
  28. 2020Contrastive Learning with Adversarial Perturbations for Conditional Text Generationcited by 56
  29. 2019Spherical Text Embeddingcited by 49
  30. 2021Learning to Retrieve Passages without Supervisioncited by 46
  31. 2020TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLPcited by 45
  32. 2022A Causal Lens for Controllable Text Generationcited by 45
  33. 2021An Investigation of the (In)effectiveness of Counterfactually Augmented Datacited by 42
  34. 2016Adversarial Diversity and Hard Positive Generationcited by 35
  35. 2021Sequence Level Contrastive Learning for Text Summarizationcited by 31
  36. 2021Improving Contrastive Learning on Imbalanced Seed Data via Open-World Samplingcited by 30
  37. 2016Nonparametric Spherical Topic Modeling with Word Embeddingscited by 16
  38. 2021Sentence Embeddings using Supervised Contrastive Learningcited by 5

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 1; 2024: 1.

5 papers.

  1. 2023ConDA: Contrastive Domain Adaptation for AI-generated Text Detectioncited by 37
  2. 2022CCC-wav2vec 2.0: Clustering aided Cross Contrastive Self-supervised learning of speech representationscited by 11
  3. 2022PromptDA: Label-guided Data Augmentation for Prompt-based Few-shot Learnerscited by 7
  4. 2024Adversarial Masking Contrastive Learning for vein recognitioncited by 1
  5. 2022Contrastive Credibility Propagation for Reliable Semi-Supervised Learningcited by 0