# SimCLAD: A Simple Framework for Contrastive Learning of Acronym Disambiguation

arXiv: https://arxiv.org/abs/2111.14306 · HTML: https://mlsanity.com/paper/2111.14306/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 29 Nov 2021.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [CLEAR: Contrastive Learning for Sentence Representation](https://mlsanity.com/paper/2012.15466/) (2020)
- [Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders](https://mlsanity.com/paper/2104.08027/) (2021)
- [ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning](https://mlsanity.com/paper/2012.15283/) (2020)
- [TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning](https://mlsanity.com/paper/2111.04198/) (2021)
- [More but Correct: Generating Diversified and Entity-revised Medical Response](https://mlsanity.com/paper/2108.01266/) (2021)
- [What Does This Acronym Mean? Introducing a New Dataset for Acronym Identification and Disambiguation](https://mlsanity.com/paper/2010.14678/) (2020)
- [Deep Contextualized Biomedical Abbreviation Expansion](https://mlsanity.com/paper/1906.03360/) (2019)
- [Acronym Identification and Disambiguation Shared Tasks for Scientific Document Understanding](https://mlsanity.com/paper/2012.11760/) (2020)
- [MadDog: A Web-based System for Acronym Identification and Disambiguation](https://mlsanity.com/paper/2101.09893/) (2021)
- [BERT-based Acronym Disambiguation with Multiple Training Strategies](https://mlsanity.com/paper/2103.00488/) (2021)
- [Leveraging Domain Agnostic and Specific Knowledge for Acronym Disambiguation](https://mlsanity.com/paper/2107.00316/) (2021)

## Built on by (most cited first, 3 of 3)

- [LingYi: Medical Conversational Question Answering System based on Multi-modal Knowledge Graphs](https://mlsanity.com/paper/2204.09220/) (2022)
- [GLADIS: A General and Large Acronym Disambiguation Benchmark](https://mlsanity.com/paper/2302.01860/) (2023)
- [Capability Localization: Capabilities Can be Localized rather than Individual Knowledge](https://mlsanity.com/paper/2502.20992/) (2025)

## Abstract (arXiv)

Acronym disambiguation means finding the correct meaning of an ambiguous acronym from the dictionary in a given sentence, which is one of the key points for scientific document understanding (SDU@AAAI-22). Recently, many attempts have tried to solve this problem via fine-tuning the pre-trained masked language models (MLMs) in order to obtain a better acronym representation. However, the acronym meaning is varied under different contexts, whose corresponding phrase representation mapped in different directions lacks discrimination in the entire vector space. Thus, the original representations of the pre-trained MLMs are not ideal for the acronym disambiguation task. In this paper, we propose a Simple framework for Contrastive Learning of Acronym Disambiguation (SimCLAD) method to better understand the acronym meanings. Specifically, we design a continual contrastive pre-training method that enhances the pre-trained model's generalization ability by learning the phrase-level contrastive distributions between true meaning and ambiguous phrases. The results on the acronym disambiguation of the scientific domain in English show that the proposed method outperforms all other competitive state-of-the-art (SOTA) methods.