# AILAB-Udine@SMM4H 22: Limits of Transformers and BERT Ensembles

arXiv: https://arxiv.org/abs/2209.03452 · HTML: https://mlsanity.com/paper/2209.03452/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 15 references that are in the map.
- First posted on arXiv 7 Sep 2022.

## Builds on (most cited first, 6 of 6)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [Improving Adverse Drug Event Extraction with SpanBERT on Different Text Typologies](https://mlsanity.com/paper/2105.08882/) (2021)

## Built on by (most cited first, 2 of 2)

- [A Dataset for Pharmacovigilance in German, French, and Japanese: Annotating Adverse Drug Reactions across Languages](https://mlsanity.com/paper/2403.18336/) (2024)
- [Identifying and Aligning Medical Claims Made on Social Media with Medical Evidence](https://mlsanity.com/paper/2405.11219/) (2024)

## Abstract (arXiv)

This paper describes the models developed by the AILAB-Udine team for the SMM4H 22 Shared Task. We explored the limits of Transformer based models on text classification, entity extraction and entity normalization, tackling Tasks 1, 2, 5, 6 and 10. The main take-aways we got from participating in different tasks are: the overwhelming positive effects of combining different architectures when using ensemble learning, and the great potential of generative models for term normalization.