# AT-BERT: Adversarial Training BERT for Acronym Identification Winning Solution for SDU@AAAI-21

arXiv: https://arxiv.org/abs/2101.03700 · HTML: https://mlsanity.com/paper/2101.03700/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 11 Jan 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Adversarial Training for Free!](https://mlsanity.com/paper/1904.12843/) (2019)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [A Survey on Deep Learning for Named Entity Recognition](https://mlsanity.com/paper/1812.09449/) (2018)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://mlsanity.com/paper/1605.07725/) (2016)
- [FreeLB: Enhanced Adversarial Training for Natural Language Understanding](https://mlsanity.com/paper/1909.11764/) (2019)
- [You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle](https://mlsanity.com/paper/1905.00877/) (2019)
- [Dice Loss for Data-imbalanced NLP Tasks](https://mlsanity.com/paper/1911.02855/) (2019)
- [What Does This Acronym Mean? Introducing a New Dataset for Acronym Identification and Disambiguation](https://mlsanity.com/paper/2010.14678/) (2020)
- [Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation](https://mlsanity.com/paper/2002.10345/) (2020)
- [Acronym Identification and Disambiguation Shared Tasks for Scientific Document Understanding](https://mlsanity.com/paper/2012.11760/) (2020)
- [Document-Level Definition Detection in Scholarly Documents: Existing Models, Error Analyses, and Future Directions](https://mlsanity.com/paper/2010.05129/) (2020)
- [Improving Slot Filling by Utilizing Contextual Information](https://mlsanity.com/paper/1911.01680/) (2019)

## Built on by (most cited first, 7 of 7)

- [GLADIS: A General and Large Acronym Disambiguation Benchmark](https://mlsanity.com/paper/2302.01860/) (2023)
- [Leveraging Domain Agnostic and Specific Knowledge for Acronym Disambiguation](https://mlsanity.com/paper/2107.00316/) (2021)
- [MACRONYM: A Large-Scale Dataset for Multilingual and Multi-Domain Acronym Extraction](https://mlsanity.com/paper/2202.09694/) (2022)
- [PLOD: An Abbreviation Detection Dataset for Scientific Documents](https://mlsanity.com/paper/2204.12061/) (2022)
- [Detecting Generated Scientific Papers using an Ensemble of Transformer Models](https://mlsanity.com/paper/2209.08283/) (2022)
- [CABACE: Injecting Character Sequence Information and Domain Knowledge for Enhanced Acronym and Long-Form Extraction](https://mlsanity.com/paper/2112.13237/) (2021)
- [An Ensemble Approach to Acronym Extraction using Transformers](https://mlsanity.com/paper/2201.03026/) (2022)

## Abstract (arXiv)

Acronym identification focuses on finding the acronyms and the phrases that have been abbreviated, which is crucial for scientific document understanding tasks. However, the limited size of manually annotated datasets hinders further improvement for the problem. Recent breakthroughs of language models pre-trained on large corpora clearly show that unsupervised pre-training can vastly improve the performance of downstream tasks. In this paper, we present an Adversarial Training BERT method named AT-BERT, our winning solution to acronym identification task for Scientific Document Understanding (SDU) Challenge of AAAI 2021. Specifically, the pre-trained BERT is adopted to capture better semantic representation. Then we incorporate the FGM adversarial training strategy into the fine-tuning of BERT, which makes the model more robust and generalized. Furthermore, an ensemble mechanism is devised to involve the representations learned from multiple BERT variants. Assembling all these components together, the experimental results on the SciAI dataset show that our proposed approach outperforms all other competitive state-of-the-art methods.