mlsanityData to 14 Oct 2026

Contextual Representation Learning beyond Masked Language Modeling

arXiv 2204.04163 ↗First posted 8 Apr 2022cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

How do masked language models (MLMs) such as BERT learn contextual representations? In this work, we analyze the learning dynamics of MLMs. We find that MLMs adopt sampled embeddings as anchors to estimate and inject contextual semantics to representations, which limits the efficiency and effectiveness of MLMs. To address these issues, we propose TACO, a simple yet effective representation learning approach to directly model global semantics. TACO extracts and aligns contextual semantics hidden in contextualized representations to encourage models to attend global semantics when generating contextualized representations. Experiments on the GLUE benchmark show that TACO achieves up to 5x speedup and up to 1.2 points average improvement over existing MLMs. The code is available at https://github.com/FUZHIYI/TACO.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 27 of its 43 references that are in the map. First posted on arXiv 8 Apr 2022.

Builds on

27 of 43 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
15 more
  1. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  2. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2018Deep contextualized word representationscited by 2,290
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  7. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  8. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  9. 2018Neural Network Acceptability Judgmentscited by 958
  10. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  11. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  12. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
  13. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  14. 2019A Mutual Information Maximization Perspective of Language Representation Learningcited by 85
  15. 2020CAPT: Contrastive Pre-Training for Learning Denoised Sequence Representationscited by 8

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 7; 2024: 1; 2025: 2.

16 papers.

Sort:

By area:

  1. 2023Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrievalcited by 132
  2. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  3. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
  4. 2023Adapting a Language Model While Preserving its General Knowledgecited by 23
  5. 2022Write and Paint: Generative Vision-Language Models are Unified Modal Learnerscited by 17
  6. 2022Multi-CLS BERT: An Efficient Alternative to Traditional Ensemblingcited by 14
  7. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Modelscited by 11
  8. 2023RARE: Robust Masked Graph Autoencodercited by 10
  9. 2022SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakescited by 9
  10. 2023Exploring the Landscape of Natural Language Processing Researchcited by 7
  11. 2023EDMAE: An Efficient Decoupled Masked Autoencoder for Standard View Identification in Pediatric Echocardiographycited by 4
  12. 2023Aerial Image Object Detection With Vision Transformer Detector (ViTDet)cited by 3
4 more
  1. 2024SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervisioncited by 3
  2. 2025ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Modelscited by 2
  3. 2023Prompt-based Logical Semantics Enhancement for Implicit Discourse Relation Recognitioncited by 1
  4. 2025Confidence Regularized Masked Language Modeling using Text Lengthcited by 0