# The futility of STILTs for the classification of lexical borrowings in Spanish

arXiv: https://arxiv.org/abs/2109.08607 · HTML: https://mlsanity.com/paper/2109.08607/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 28 references that are in the map.
- First posted on arXiv 17 Sep 2021.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks](https://mlsanity.com/paper/1811.01088/) (2018)
- [Exploring and Predicting Transferability across NLP Tasks](https://mlsanity.com/paper/2005.00770/) (2020)
- [LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation](https://mlsanity.com/paper/2005.04322/) (2020)
- [What to Pre-Train on? Efficient Intermediate Task Selection](https://mlsanity.com/paper/2104.08247/) (2021)
- [Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigation](https://mlsanity.com/paper/2008.06788/) (2020)
- [Load What You Need: Smaller Versions of Multilingual BERT](https://mlsanity.com/paper/2010.05609/) (2020)

## Built on by (most cited first, 6 of 6)

- [Detecting Unassimilated Borrowings in Spanish: An Annotated Corpus and Approaches to Modeling](https://mlsanity.com/paper/2203.16169/) (2022)
- [Overview of ADoBo 2021: Automatic Detection of Unassimilated Borrowings in the Spanish Press](https://mlsanity.com/paper/2110.15682/) (2021)
- [Overview of ADoBo at IberLEF 2025: Automatic Detection of Anglicisms in Spanish](https://mlsanity.com/paper/2507.21813/) (2025)
- [LuxBorrow: From Pompier to Pompjee, Tracing Borrowing in Luxembourgish](https://mlsanity.com/paper/2603.10789/) (2026)
- [Observatorio Lazaro: A self-populating database of anglicism usage in the Spanish press](https://mlsanity.com/paper/2608.00713/) (2026)
- [pylazaro: a Python package for anglicism extraction in Spanish](https://mlsanity.com/paper/2609.29276/) (2026)

## Abstract (arXiv)

The first edition of the IberLEF 2021 shared task on automatic detection of borrowings (ADoBo) focused on detecting lexical borrowings that appeared in the Spanish press and that have recently been imported into the Spanish language. In this work, we tested supplementary training on intermediate labeled-data tasks (STILTs) from part of speech (POS), named entity recognition (NER), code-switching, and language identification approaches to the classification of borrowings at the token level using existing pre-trained transformer-based language models. Our extensive experimental results suggest that STILTs do not provide any improvement over direct fine-tuning of multilingual models. However, multilingual models trained on small subsets of languages perform reasonably better than multilingual BERT but not as good as multilingual RoBERTa for the given dataset.