# JuniperLiu at CoMeDi Shared Task: Models as Annotators in Lexical Semantics Disagreements

arXiv: https://arxiv.org/abs/2411.12147 · HTML: https://mlsanity.com/paper/2411.12147/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 19 Nov 2024.

## Builds on (most cited first, 17 of 17)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Probing Pretrained Language Models for Lexical Semantics](https://mlsanity.com/paper/2010.05731/) (2020)
- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics](https://mlsanity.com/paper/2403.01509/) (2024)
- [Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change](https://mlsanity.com/paper/1804.06517/) (2018)
- [DWUG: A large Resource of Diachronic Word Usage Graphs in Four Languages](https://mlsanity.com/paper/2104.08540/) (2021)
- [RuSemShift: a dataset of historical lexical semantic change in Russian](https://mlsanity.com/paper/2010.06436/) (2020)
- [Lexical Semantic Change Discovery](https://mlsanity.com/paper/2106.03111/) (2021)
- [RuDSI: graph-based word sense induction dataset for Russian](https://mlsanity.com/paper/2209.13750/) (2022)

## Abstract (arXiv)

We present the results of our system for the CoMeDi Shared Task, which predicts majority votes (Subtask 1) and annotator disagreements (Subtask 2). Our approach combines model ensemble strategies with MLP-based and threshold-based methods trained on pretrained language models. Treating individual models as virtual annotators, we simulate the annotation process by designing aggregation measures that incorporate continuous relatedness scores and discrete classification labels to capture both majority and disagreement. Additionally, we employ anisotropy removal techniques to enhance performance. Experimental results demonstrate the effectiveness of our methods, particularly for Subtask 2. Notably, we find that standard deviation on continuous relatedness scores among different model manipulations correlates with human disagreement annotations compared to metrics on aggregated discrete labels. The code will be published at https://github.com/RyanLiut/CoMeDi_Solution.