# Combining Contrastive Learning and Knowledge Graph Embeddings to develop medical word embeddings for the Italian language

arXiv: https://arxiv.org/abs/2211.05035 · HTML: https://mlsanity.com/paper/2211.05035/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 9 Nov 2022.

## Builds on (most cited first, 15 of 15)

- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [Self-Alignment Pretraining for Biomedical Entity Representations](https://mlsanity.com/paper/2010.11784/) (2020)
- [Episodic Memory in Lifelong Language Learning](https://mlsanity.com/paper/1906.01076/) (2019)
- [CODER: Knowledge infused cross-lingual medical term embedding for term normalization](https://mlsanity.com/paper/2011.02947/) (2020)
- [Improving Biomedical Pretrained Language Models with Knowledge](https://mlsanity.com/paper/2104.10344/) (2021)
- [Learning Domain-Specialised Representations for Cross-Lingual Biomedical Entity Linking](https://mlsanity.com/paper/2105.14398/) (2021)
- [Automatic Biomedical Term Clustering by Learning Fine-grained Term Representations](https://mlsanity.com/paper/2204.00391/) (2022)
- [The General Pair-based Weighting Loss for Deep Metric Learning](https://mlsanity.com/paper/1905.12837/) (2019)

## Abstract (arXiv)

Word embeddings play a significant role in today's Natural Language Processing tasks and applications. While pre-trained models may be directly employed and integrated into existing pipelines, they are often fine-tuned to better fit with specific languages or domains. In this paper, we attempt to improve available embeddings in the uncovered niche of the Italian medical domain through the combination of Contrastive Learning (CL) and Knowledge Graph Embedding (KGE). The main objective is to improve the accuracy of semantic similarity between medical terms, which is also used as an evaluation task. Since the Italian language lacks medical texts and controlled vocabularies, we have developed a specific solution by combining preexisting CL methods (multi-similarity loss, contextualization, dynamic sampling) and the integration of KGEs, creating a new variant of the loss. Although without having outperformed the state-of-the-art, represented by multilingual models, the obtained results are encouraging, providing a significant leap in performance compared to the starting model, while using a significantly lower amount of data.