mlsanityData to 7 Oct 2026

MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models

arXiv 2109.09237 ↗First posted 19 Sep 2021cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

Recent work indicated that pretrained language models (PLMs) such as BERT and RoBERTa can be transformed into effective sentence and word encoders even via simple self-supervised techniques. Inspired by this line of work, in this paper we propose a fully unsupervised approach to improving word-in-context (WiC) representations in PLMs, achieved via a simple and efficient WiC-targeted fine-tuning procedure: MirrorWiC. The proposed method leverages only raw texts sampled from Wikipedia, assuming no sense-annotated data, and learns context-aware word representations within a standard contrastive learning setup. We experiment with a series of standard and comprehensive WiC benchmarks across multiple languages. Our proposed fully unsupervised MirrorWiC models obtain substantial gains over off-the-shelf PLMs across all monolingual, multilingual and cross-lingual setups. Moreover, on some standard WiC benchmarks, MirrorWiC is even on-par with supervised models fine-tuned with in-task data and sense labels.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 30 of its 47 references that are in the map. First posted on arXiv 19 Sep 2021.

Builds on

30 of 47 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  8. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  9. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  10. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  11. 2020Self-Alignment Pretraining for Biomedical Entity Representationscited by 233
  12. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
18 more
  1. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  2. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  3. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  4. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  5. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  6. 2019SenseBERT: Driving Some Sense into BERTcited by 73
  7. 2020Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoderscited by 61
  8. 2019Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddingscited by 55
  9. 2018From Word to Sense Embeddings: A Survey on Vector Representations of Meaningcited by 49
  10. 2020XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualizationcited by 42
  11. 2019Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguationcited by 40
  12. 2021Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Sensescited by 39
  13. 2021A Cluster-based Approach for Improving Isotropy in Contextual Embedding Spacecited by 27
  14. 2020Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillationcited by 25
  15. 2019Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representationscited by 21
  16. 2021AM2iCo: Evaluating Word Meaning in Context across Low-Resource Languages with Adversarial Examplescited by 14
  17. 2019CoSimLex: A Resource for Evaluating Graded Word Similarity in Contextcited by 13
  18. 2020WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Contextcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 2; 2023: 4; 2024: 5; 2025: 3; 2026: 2.

18 papers.

Sort:

  1. 2021TaCL: Improving BERT Pre-training with Token-aware Contrastive Learningcited by 32
  2. 2021Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Modelscited by 21
  3. 2022Improving Word Translation via Two-Stage Contrastive Learningcited by 18
  4. 2023Distilling Semantic Concept Embeddings from Contrastively Fine-Tuned Language Modelscited by 8
  5. 2024Ontology Completion with Natural Language Inference and Concept Embeddings: An Analysiscited by 4
  6. 2023EnCore: Fine-Grained Entity Typing by Pre-Training Entity Encoders on Coreference Chainscited by 2
  7. 2026In the LLM era, Word Sense Induction remains unsolvedcited by 2
  8. 2023Labels Need Prompts Too: Mask Matching for Natural Language Understanding Taskscited by 1
  9. 2024Injecting Wiktionary to improve token-level contextual representations using contrastive learningcited by 1
  10. 2022Reranking Overgenerated Responses for End-to-End Task-Oriented Dialogue Systemscited by 0
  11. 2023Entity or Relation Embeddings? An Analysis of Encoding Strategies for Relation Extractioncited by 0
  12. 2024Modelling Commonsense Commonalities with Multi-Facet Concept Embeddingscited by 0
6 more
  1. 2024Distilling Monolingual and Crosslingual Word-in-Context Representationscited by 0
  2. 2024Modeling Multi-modal Cross-interaction for Multi-label Few-shot Image Classification Based on Local Feature Selectioncited by 0
  3. 2025Automatically Suggesting Diverse Example Sentences for L2 Japanese Learners Using Pre-Trained Language Modelscited by 0
  4. 2025CALE : Concept-Aligned Embeddings for Both Within-Lemma and Inter-Lemma Sense Differentiationcited by 0
  5. 2025Extracting Conceptual Spaces from LLMs Using Prototype Embeddingscited by 0
  6. 2026A graph-based analysis of semantic types and coercion in contextualized word embeddingscited by 0