# XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond

arXiv: https://arxiv.org/abs/2104.12250 · HTML: https://mlsanity.com/paper/2104.12250/

## Key facts

- Cited by 114 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 45 references that are in the map.
- First posted on arXiv 25 Apr 2021.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [Learning Word Vectors for 157 Languages](https://mlsanity.com/paper/1802.06893/) (2018)
- [SemEval-2017 Task 4: Sentiment Analysis in Twitter](https://mlsanity.com/paper/1912.00741/) (2019)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets](https://mlsanity.com/paper/2008.04277/) (2020)
- [Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classification](https://mlsanity.com/paper/1606.01614/) (2016)
- [An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models](https://mlsanity.com/paper/1902.10547/) (2019)
- [Building a Sentiment Corpus of Tweets in Brazilian Portuguese](https://mlsanity.com/paper/1712.08917/) (2017)
- [Sentiment analysis is not solved! Assessing and probing sentiment classification](https://mlsanity.com/paper/1906.05887/) (2019)
- [Improving Sentiment Analysis over non-English Tweets using Multilingual Transformers and Automatic Translation for Data-Augmentation](https://mlsanity.com/paper/2010.03486/) (2020)

## Built on by (most cited first, 30 of 114)

- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [MAGE: Machine-generated Text Detection in the Wild](https://mlsanity.com/paper/2305.13242/) (2023)
- [TweetNLP: Cutting-Edge Natural Language Processing for Social Media](https://mlsanity.com/paper/2206.14774/) (2022)
- [AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages](https://mlsanity.com/paper/2302.08956/) (2023)
- [LLMs Are Few-Shot In-Context Low-Resource Language Learners](https://mlsanity.com/paper/2403.16512/) (2024)
- [ChatGPT: A Meta-Analysis after 2.5 Months](https://mlsanity.com/paper/2302.13795/) (2023)
- [Reasoning Runtime Behavior of a Program with LLM: How Far Are We?](https://mlsanity.com/paper/2403.16437/) (2024)
- [SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)](https://mlsanity.com/paper/2304.06845/) (2023)
- [Overview of AuTexTification at IberLEF 2023: Detection and Attribution of Machine-Generated Text in Multiple Domains](https://mlsanity.com/paper/2309.11285/) (2023)
- [An Efficient Multilingual Language Model Compression through Vocabulary Trimming](https://mlsanity.com/paper/2305.15020/) (2023)
- [Zero- and Few-Shot Prompting with LLMs: A Comparative Study with Fine-tuned Models for Bangla Sentiment Analysis](https://mlsanity.com/paper/2308.10783/) (2023)
- [Unsupervised Layer-wise Score Aggregation for Textual OOD Detection](https://mlsanity.com/paper/2302.09852/) (2023)
- [What Drives Performance in Multilingual Language Models?](https://mlsanity.com/paper/2404.19159/) (2024)
- [HERDPhobia: A Dataset for Hate Speech against Fulani in Nigeria](https://mlsanity.com/paper/2211.15262/) (2022)
- [ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing](https://mlsanity.com/paper/2310.11166/) (2023)
- [Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners](https://mlsanity.com/paper/2405.13816/) (2024)
- [NaijaHate: Evaluating Hate Speech Detection on Nigerian Twitter Using Representative Data](https://mlsanity.com/paper/2403.19260/) (2024)
- [mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humans](https://mlsanity.com/paper/2406.04215/) (2024)
- [pysentimiento: A Python Toolkit for Opinion Mining and Social NLP tasks](https://mlsanity.com/paper/2106.09462/) (2021)
- [ArmanEmo: A Persian Dataset for Text-based Emotion Detection](https://mlsanity.com/paper/2207.11808/) (2022)
- [SuperTweetEval: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research](https://mlsanity.com/paper/2310.14757/) (2023)
- [Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognition](https://mlsanity.com/paper/2310.18877/) (2023)
- [UZH_CLyp at SemEval-2023 Task 9: Head-First Fine-Tuning and ChatGPT Data Generation for Cross-Lingual Learning in Tweet Intimacy Prediction](https://mlsanity.com/paper/2303.01194/) (2023)
- [Uncovering Political Hate Speech During Indian Election Campaign: A New Low-Resource Dataset and Baselines](https://mlsanity.com/paper/2306.14764/) (2023)
- [Data-Efficient Strategies for Expanding Hate Speech Detection into Under-Resourced Languages](https://mlsanity.com/paper/2210.11359/) (2022)
- [Using Emotion Embeddings to Transfer Knowledge Between Emotions, Languages, and Annotation Formats](https://mlsanity.com/paper/2211.00171/) (2022)
- [Diversity-oriented Data Augmentation with Large Language Models](https://mlsanity.com/paper/2502.11671/) (2025)
- [Can Prompting LLMs Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Study](https://mlsanity.com/paper/2505.06149/) (2025)
- [HUME: Measuring the Human-Model Performance Gap in Text Embedding Tasks](https://mlsanity.com/paper/2510.10062/) (2025)
- [Don't Take This Out of Context! On the Need for Contextual Models and Evaluations for Stylistic Rewriting](https://mlsanity.com/paper/2305.14755/) (2023)

## Abstract (arXiv)

Language models are ubiquitous in current NLP, and their multilingual capacity has recently attracted considerable attention. However, current analyses have almost exclusively focused on (multilingual variants of) standard benchmarks, and have relied on clean pre-training and task-specific corpora as multilingual signals. In this paper, we introduce XLM-T, a model to train and evaluate multilingual language models in Twitter. In this paper we provide: (1) a new strong multilingual baseline consisting of an XLM-R (Conneau et al. 2020) model pre-trained on millions of tweets in over thirty languages, alongside starter code to subsequently fine-tune on a target task; and (2) a set of unified sentiment analysis Twitter datasets in eight different languages and a XLM-T model fine-tuned on them.