# drsphelps at SemEval-2022 Task 2: Learning idiom representations using BERTRAM

arXiv: https://arxiv.org/abs/2204.02821 · HTML: https://mlsanity.com/paper/2204.02821/

## Key facts

- Cited by 5 papers in the map (data through 7 Oct 2026).
- Builds on 11 of its 20 references that are in the map.
- First posted on arXiv 6 Apr 2022.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Mimicking Word Embeddings using Subword RNNs](https://mlsanity.com/paper/1707.06961/) (2017)
- [AStitchInLanguageModels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language Models](https://mlsanity.com/paper/2109.04413/) (2021)
- [BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance](https://mlsanity.com/paper/1910.07181/) (2019)
- [Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts](https://mlsanity.com/paper/1904.01617/) (2019)
- [Generalizing Word Embeddings using Bag of Subwords](https://mlsanity.com/paper/1809.04259/) (2018)
- [Bertinho: Galician BERT Representations](https://mlsanity.com/paper/2103.13799/) (2021)

## Built on by (most cited first, 5 of 5)

- [SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding](https://mlsanity.com/paper/2204.10050/) (2022)
- [Enhancing Idiomatic Representation in Multiple Languages via an Adaptive Contrastive Triplet Loss](https://mlsanity.com/paper/2406.15175/) (2024)
- [Sample Efficient Approaches for Idiomaticity Detection](https://mlsanity.com/paper/2205.11306/) (2022)
- [Is Less More? Quality, Quantity and Context in Idiom Processing with Natural Language Models](https://mlsanity.com/paper/2405.08497/) (2024)
- [Unsupervised Paraphrasing of Multiword Expressions](https://mlsanity.com/paper/2306.01443/) (2023)

## Abstract (arXiv)

This paper describes our system for SemEval-2022 Task 2 Multilingual Idiomaticity Detection and Sentence Embedding sub-task B. We modify a standard BERT sentence transformer by adding embeddings for each idioms, which are created using BERTRAM and a small number of contexts. We show that this technique increases the quality of idiom representations and leads to better performance on the task. We also perform analysis on our final results and show that the quality of the produced idiom embeddings is highly sensitive to the quality of the input contexts.