mlsanityData to 7 Oct 2026

Using Deep Mixture-of-Experts to Detect Word Meaning Shift for TempoWiC

arXiv 2211.03466 ↗First posted 7 Nov 2022cs.CLcited by 2

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

This paper mainly describes the dma submission to the TempoWiC task, which achieves a macro-F1 score of 77.05% and attains the first place in this task. We first explore the impact of different pre-trained language models. Then we adopt data cleaning, data augmentation, and adversarial training strategies to enhance the model generalization and robustness. For further improvement, we integrate POS information and word semantic representation using a Mixture-of-Experts (MoE) approach. The experimental results show that MoE can overcome the feature overuse issue and combine the context, POS, and word semantic features well. Additionally, we use a model ensemble method for the final prediction, which has been proven effective by many research works.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 14 of its 17 references that are in the map. First posted on arXiv 7 Nov 2022.

Builds on

14 of 17 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 13

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  6. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  7. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  8. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  9. 2020XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualizationcited by 42
  10. 2021Explaining and Improving BERT Performance on Lexical Semantic Change Detectioncited by 33
  11. 2022TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Mediacited by 19
  12. 2021Statistically significant detection of semantic shifts using contextual word embeddingscited by 14
1 more
  1. 2019Tree-gated Deep Mixture-of-Experts For Pose-robust Face Alignmentcited by 5

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024(Chat)GPT v BERT: Dawn of Justice for Semantic Change Detectioncited by 7
  2. 2024OPDAI at SemEval-2024 Task 6: Small LLMs can Accelerate Hallucination Detection with Weakly Supervised Datacited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Mixture of Experts in Large Language Models
  2. 2025Mixture of Experts in Large Language Models
  3. 2023Memory Augmented Language Models through Mixture of Word Experts
  4. 2021Efficient Large Scale Language Modeling with Mixtures of Experts
  5. 2024T-REX: Mixture-of-Rank-One-Experts with Semantic-aware Intuition for Multi-task Large Language Model Finetuning
  6. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
  7. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  8. 2023Bidirectional Attention as a Mixture of Continuous Word Experts