mlsanityData to 7 Oct 2026

COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

arXiv 2102.08473 ↗First posted 16 Feb 2021cs.CL cs.LGcited by 126

Not on the map. Why?

Abstract (arXiv)

We present a self-supervised learning framework, COCO-LM, that pretrains Language Models by COrrecting and COntrasting corrupted text sequences. Following ELECTRA-style pretraining, COCO-LM employs an auxiliary language model to corrupt text sequences, upon which it constructs two new tasks for pretraining the main model. The first token-level task, Corrective Language Modeling, is to detect and correct tokens replaced by the auxiliary model, in order to better capture token-level semantics. The second sequence-level task, Sequence Contrastive Learning, is to align text sequences originated from the same source input while ensuring uniformity in the representation space. Experiments on GLUE and SQuAD demonstrate that COCO-LM not only outperforms recent state-of-the-art pretrained models in accuracy, but also improves pretraining efficiency. It achieves the MNLI accuracy of ELECTRA with 50% of its pretraining GPU hours. With the same pretraining steps of standard base/large-sized models, COCO-LM outperforms the previous best models by 1+ GLUE average points.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Builds on 57 of its 75 references that are in the map. First posted on arXiv 16 Feb 2021.

Builds on

57 of 75 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
38 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  13. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  14. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  15. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  16. 2018Neural Network Acceptability Judgmentscited by 952
  17. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  18. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  19. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  20. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  21. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  22. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  23. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  24. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  25. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 275
  26. 2020Transformers as Soft Reasoners over Languagecited by 265
  27. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
  28. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  29. 2018A Simple Method for Commonsense Reasoningcited by 223
  30. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  31. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 213
  32. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  33. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  34. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  35. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  36. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  37. 2020Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biasescited by 143
  38. 2020Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Taskscited by 136

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 23; 2022: 49; 2023: 26; 2024: 18; 2025: 7; 2026: 3.

126 papers.

Sort:

By area:

  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  2. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  3. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  4. 2023A Cookbook of Self-Supervised Learningcited by 248
  5. 2022A Contrastive Framework for Neural Text Generationcited by 224
  6. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  7. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 190
  8. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  9. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  10. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  11. 2021Supporting Clustering with Contrastive Learningcited by 90
  12. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
38 more
  1. 2021ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embeddingcited by 73
  2. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
  3. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  4. 2022Corrupted Image Modeling for Self-Supervised Visual Pre-Trainingcited by 68
  5. 2022Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classificationcited by 52
  6. 2023Patton: Language Model Pretraining on Text-Rich Networkscited by 49
  7. 2022TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twittercited by 44
  8. 2021Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classificationcited by 43
  9. 2021A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectivescited by 41
  10. 2021Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Trainingcited by 41
  11. 2021Improved Text Classification via Contrastive Adversarial Trainingcited by 38
  12. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Freecited by 37
  13. 2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
  14. 2022Topic Discovery via Latent Space Clustering of Pretrained Language Model Representationscited by 34
  15. 2022UCTopic: Unsupervised Contrastive Learning for Phrase Representations and Topic Miningcited by 33
  16. 2021TaCL: Improving BERT Pre-training with Token-aware Contrastive Learningcited by 32
  17. 2021Pairwise Supervised Contrastive Learning of Sentence Representationscited by 31
  18. 2024Generative AI for Synthetic Data Generation: Methods, Challenges and the Futurecited by 31
  19. 2022CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Groundingcited by 27
  20. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  21. 2022English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddingscited by 26
  22. 2022InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddingscited by 25
  23. 2021Virtual Augmentation Supported Contrastive Learning of Sentence Representationscited by 21
  24. 2022CoCoSoDa: Effective Contrastive Learning for Code Searchcited by 21
  25. 2022EASE: Entity-Aware Contrastive Learning of Sentence Embeddingcited by 21
  26. 2022METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signalscited by 20
  27. 2022DICE: Data-Efficient Clinical Event Extraction with Generative Modelscited by 20
  28. 2023Language Models As Semantic Indexerscited by 20
  29. 2021DialogueCSE: Dialogue-based Contrastive Learning of Sentence Embeddingscited by 19
  30. 2022COCO-DR: Combating Distribution Shifts in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learningcited by 19
  31. 2022ClueWeb22: 10 Billion Web Documents with Visual and Semantic Informationcited by 19
  32. 2024Customizing Language Model Responses with Contrastive In-Context Learningcited by 19
  33. 2022A Simple Contrastive Learning Objective for Alleviating Neural Text Degenerationcited by 18
  34. 2022ContraCLM: Contrastive Learning For Causal Language Modelcited by 18
  35. 2022Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negativescited by 16
  36. 2022Beyond English-Centric Bitexts for Better Multilingual Language Representation Learningcited by 16
  37. 2023Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learningcited by 16
  38. 2024HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Predictioncited by 15

50 most cited of 126 All 126 (JSON)