mlsanityData to 7 Oct 2026

Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning

arXiv 2109.05941 ↗First posted 10 Sep 2021cs.CL cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

We introduce EfficientCL, a memory-efficient continual pretraining method that applies contrastive learning with novel data augmentation and curriculum learning. For data augmentation, we stack two types of operation sequentially: cutoff and PCA jittering. While pretraining steps proceed, we apply curriculum learning by incrementing the augmentation degree for each difficulty step. After data augmentation is finished, contrastive learning is applied on projected embeddings of original and augmented examples. When finetuned on GLUE benchmark, our model outperforms baseline models, especially for sentence-level tasks. Additionally, this improvement is capable with only 70% of computational memory compared to the baseline model.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 19 of its 24 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

19 of 24 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  6. 2020Contrastive Learning with Hard Negative Samplescited by 619
  7. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  8. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 285
  9. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 275
  10. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  11. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  12. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
7 more
  1. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  2. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  3. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  4. 2020A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generationcited by 72
  5. 2020CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understandingcited by 41
  6. 2019An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Modelscited by 39
  7. 2021Few-Shot Text Classification with Triplet Networks, Data Augmentation, and Curriculum Learningcited by 35

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 2; 2024: 8; 2025: 1.

14 papers.

Sort:

By area:

  1. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  2. 2023Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relationscited by 22
  3. 2024LLM-R2: A Large Language Model Enhanced Rule-based Rewrite System for Boosting Query Efficiencycited by 16
  4. 2025Efficient Self-Supervised Learning for Earth Observation via Dynamic Dataset Curationcited by 7
  5. 2024Contrastive Learning with Synthetic Positivescited by 6
  6. 2024Diffusion Curriculum: Synthetic-to-Real Data Curriculum via Image-Guided Diffusioncited by 6
  7. 2022Label-Driven Denoising Framework for Multi-Label Few-Shot Aspect Category Detectioncited by 5
  8. 2022A Curriculum Learning Approach for Multi-domain Text Classification Using Keyword weight Rankingcited by 2
  9. 2024Unsupervised hard Negative Augmentation for contrastive learningcited by 2
  10. 2024Colorful Cutout: Enhancing Image Data Augmentation with Curriculum Learningcited by 2
  11. 2022ConEntail: An Entailment-based Framework for Universal Zero and Few Shot Classification with Supervised Contrastive Pretrainingcited by 0
  12. 2023Data Augmentation for Improving Tail-traffic Robustness in Skill-routing for Dialogue Systemscited by 0
2 more
  1. 2024DNABERT-S: Pioneering Species Differentiation with Species-Aware DNA Embeddingscited by 0
  2. 2024Adverb Is the Key: Simple Text Data Augmentation with Adverb Deletioncited by 0