mlsanityData to 7 Oct 2026

TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning

arXiv 2111.04198 ↗First posted 7 Nov 2021cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Masked language models (MLMs) such as BERT and RoBERTa have revolutionized the field of Natural Language Understanding in the past few years. However, existing pre-trained MLMs often output an anisotropic distribution of token representations that occupies a narrow subset of the entire representation space. Such token representations are not ideal, especially for tasks that demand discriminative semantic meanings of distinct tokens. In this work, we propose TaCL (Token-aware Contrastive Learning), a novel continual pre-training approach that encourages BERT to learn an isotropic and discriminative distribution of token representations. TaCL is fully unsupervised and requires no additional data. We extensively test our approach on a wide range of English and Chinese benchmarks. The results show that TaCL brings consistent and notable improvements over the original BERT model. Furthermore, we conduct detailed analysis to reveal the merits and inner-workings of our approach.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 46 of its 73 references that are in the map. First posted on arXiv 7 Nov 2021.

Builds on

46 of 73 references.

References · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
34 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  5. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  6. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  7. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  8. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  9. 2020Self-Alignment Pretraining for Biomedical Entity Representationscited by 233
  10. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  11. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  12. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  13. 2021Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue Systemcited by 139
  14. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 131
  15. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  16. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  17. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  18. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  19. 2018Chinese NER Using Lattice LSTMcited by 81
  20. 2021DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarizationcited by 72
  21. 2020FLAT: Chinese NER Using Flat-Lattice Transformercited by 67
  22. 2021CONTaiNER: Few-Shot Named Entity Recognition via Contrastive Learningcited by 64
  23. 2021Plan-then-Generate: Controlled Data-to-Text Generation via Planningcited by 52
  24. 2019Glyce: Glyph-vectors for Chinese Character Representationscited by 46
  25. 2021Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Explorationcited by 44
  26. 2021Non-Autoregressive Text Generation with Pre-trained Language Modelscited by 36
  27. 2020Multilingual BERT Post-Pretraining Alignmentcited by 30
  28. 2020Dialogue Response Selection with Hierarchical Curriculum Learningcited by 27
  29. 2020Prototype-to-Style: Dialogue Generation with Style-Aware Editing on Retrieval Memorycited by 22
  30. 2021Few-Shot Table-to-Text Generation with Prototype Memorycited by 21
  31. 2021Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Modelscited by 21
  32. 2021MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Modelscited by 18
  33. 2016F-Score Driven Max Margin Neural Network for Named Entity Recognition in Chinese Social Mediacited by 11
  34. 2021Exploring Dense Retrieval for Dialogue Response Selectioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 18; 2023: 8; 2024: 3; 2025: 1; 2026: 1.

32 papers.

Sort:

  1. 2022A Contrastive Framework for Neural Text Generationcited by 224
  2. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  3. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
  4. 2022Contrastive Search Is What You Need For Neural Text Generationcited by 58
  5. 2022Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulationscited by 44
  6. 2022Nonparametric Masked Language Modelingcited by 39
  7. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 38
  8. 2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
  9. 2022CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Data Limitation With Contrastive Learningcited by 32
  10. 2022Modality-Aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detectioncited by 26
  11. 2023Adapting a Language Model While Preserving its General Knowledgecited by 23
  12. 2022On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learningcited by 15
20 more
  1. 2022HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbationcited by 11
  2. 2022R2D2: Robust Data-to-Text with Replacement Detectioncited by 8
  3. 2022Effidit: Your AI Writing Assistantcited by 8
  4. 2022From Easy to Hard: A Dual Curriculum Learning Framework for Context-Aware Document Rankingcited by 5
  5. 2023Headless Language Models: Learning without Predicting with Contrastive Weight Tyingcited by 5
  6. 2023SCStory: Self-supervised and Continual Online Story Discoverycited by 5
  7. 2025Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learningcited by 5
  8. 2022SDCL: Self-Distillation Contrastive Learning for Chinese Spell Checkingcited by 4
  9. 2023Transfer-Free Data-Efficient Multilingual Slot Labelingcited by 4
  10. 2021SimCLAD: A Simple Framework for Contrastive Learning of Acronym Disambiguationcited by 3
  11. 2022Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Modelcited by 3
  12. 2022A Benchmark Study of Contrastive Learning for Arabic Social Meaningcited by 2
  13. 2022Self-supervised Rewiring of Pre-trained Speech Encoders: Towards Faster Fine-tuning with Less Labels in Speech Processingcited by 2
  14. 2023Weighted Sampling for Masked Language Modelingcited by 2
  15. 2022Learning Locality and Isotropy in Dialogue Modelingcited by 1
  16. 2023Sentence-Level Relation Extraction via Contrastive Learning with Descriptive Relation Promptscited by 1
  17. 2024Injecting Wiktionary to improve token-level contextual representations using contrastive learningcited by 1
  18. 2024Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understandingcited by 0
  19. 2024Modeling Comparative Logical Relation with Contrastive Learning for Text Generationcited by 0
  20. 2026SIGMA: Semantic Identifier Grouping for Molecular Autoregressioncited by 0