# Boosting Transformers for Job Expression Extraction and Classification in a Low-Resource Setting

arXiv: https://arxiv.org/abs/2109.08597 · HTML: https://mlsanity.com/paper/2109.08597/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 17 Sep 2021.

## Builds on (most cited first, 10 of 10)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://mlsanity.com/paper/2010.12309/) (2020)
- [We Need to Talk About Random Splits](https://mlsanity.com/paper/2005.00636/) (2020)
- [The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain](https://mlsanity.com/paper/2006.03039/) (2020)
- [To Share or not to Share: Predicting Sets of Sources for Model Transfer Learning](https://mlsanity.com/paper/2104.08078/) (2021)
- [NLNDE: The Neither-Language-Nor-Domain-Experts' Way of Spanish Medical Document De-Identification](https://mlsanity.com/paper/2007.01030/) (2020)
- [NLNDE at CANTEMIST: Neural Sequence Labeling and Parsing Approaches for Clinical Concept Extraction](https://mlsanity.com/paper/2010.12322/) (2020)
- [NLNDE: Enhancing Neural Sequence Taggers with Attention and Noisy Channel for Robust Pharmacological Entity Detection](https://mlsanity.com/paper/2007.01022/) (2020)

## Built on by (most cited first, 1 of 1)

- [CLIN-X: pre-trained language models and a study on cross-task transfer for concept extraction in the clinical domain](https://mlsanity.com/paper/2112.08754/) (2021)

## Abstract (arXiv)

In this paper, we explore possible improvements of transformer models in a low-resource setting. In particular, we present our approaches to tackle the first two of three subtasks of the MEDDOPROF competition, i.e., the extraction and classification of job expressions in Spanish clinical texts. As neither language nor domain experts, we experiment with the multilingual XLM-R transformer model and tackle these low-resource information extraction tasks as sequence-labeling problems. We explore domain- and language-adaptive pretraining, transfer learning and strategic datasplits to boost the transformer model. Our results show strong improvements using these methods by up to 5.3 F1 points compared to a fine-tuned XLM-R model. Our best models achieve 83.2 and 79.3 F1 for the first two tasks, respectively.