Zhestyatsky at SemEval-2021 Task 2: ReLU over Cosine Similarity for BERT Fine-tuning
Not on the map. Why?
Abstract (arXiv)
This paper presents our contribution to SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation (MCL-WiC). Our experiments cover English (EN-EN) sub-track from the multilingual setting of the task. We experiment with several pre-trained language models and investigate an impact of different top-layers on fine-tuning. We find the combination of Cosine Similarity and ReLU activation leading to the most effective fine-tuning procedure. Our best model results in accuracy 92.7%, which is the fourth-best score in EN-EN sub-track.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 9 of its 12 references that are in the map. First posted on arXiv 13 Apr 2021.
Builds on
9 of 12 references.
References · 9
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
- 2018Deep contextualized word representationscited by 2,285
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
- 2016Enriching Word Vectors with Subword Informationcited by 1,578
- 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
- 2018Universal Language Model Fine-tuning for Text Classificationcited by 50
Built on by
Citing papers per year: 2026: 1.
1 paper.