# NLNDE at SemEval-2023 Task 12: Adaptive Pretraining and Source Language Selection for Low-Resource Multilingual Sentiment Analysis

arXiv: https://arxiv.org/abs/2305.00090 · HTML: https://mlsanity.com/paper/2305.00090/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 28 Apr 2023.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://mlsanity.com/paper/2010.12309/) (2020)
- [Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning](https://mlsanity.com/paper/2204.06487/) (2022)
- [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019)
- [AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages](https://mlsanity.com/paper/2302.08956/) (2023)
- [Learning to select data for transfer learning with Bayesian Optimization](https://mlsanity.com/paper/1707.05246/) (2017)
- [AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages](https://mlsanity.com/paper/2211.03263/) (2022)
- [SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)](https://mlsanity.com/paper/2304.06845/) (2023)
- [MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition](https://mlsanity.com/paper/2210.12391/) (2022)
- [Forward-Backward Selection with Early Dropping](https://mlsanity.com/paper/1705.10770/) (2017)
- [To Share or not to Share: Predicting Sets of Sources for Model Transfer Learning](https://mlsanity.com/paper/2104.08078/) (2021)
- [CLIN-X: pre-trained language models and a study on cross-task transfer for concept extraction in the clinical domain](https://mlsanity.com/paper/2112.08754/) (2021)
- [Geographical Distance Is The New Hyperparameter: A Case Study Of Finding The Optimal Pre-trained Language For English-isiZulu Machine Translation](https://mlsanity.com/paper/2205.08621/) (2022)

## Built on by (most cited first, 10 of 10)

- [Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages](https://mlsanity.com/paper/2305.12182/) (2023)
- [Goldfish: Monolingual Language Models for 350 Languages](https://mlsanity.com/paper/2408.10441/) (2024)
- [Rehearsal-Free Modular and Compositional Continual Learning for Language Models](https://mlsanity.com/paper/2404.00790/) (2024)
- [EthioLLM: Multilingual Large Language Models for Ethiopian Languages with Task Evaluation](https://mlsanity.com/paper/2403.13737/) (2024)
- [TADA: Efficient Task-Agnostic Domain Adaptation for Transformers](https://mlsanity.com/paper/2305.12717/) (2023)
- [GradSim: Gradient-Based Language Grouping for Effective Multilingual Training](https://mlsanity.com/paper/2310.15269/) (2023)
- [XAMPLER: Learning to Retrieve Cross-Lingual In-Context Examples](https://mlsanity.com/paper/2405.05116/) (2024)
- [AfroXLMR-Social: Adapting Pre-trained Language Models for African Languages Social Media Text](https://mlsanity.com/paper/2503.18247/) (2025)
- [LowResource at BLP-2023 Task 2: Leveraging BanglaBert for Low Resource Sentiment Analysis of Bangla Language](https://mlsanity.com/paper/2311.12735/) (2023)
- [mPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2305.13684/) (2023)

## Abstract (arXiv)

This paper describes our system developed for the SemEval-2023 Task 12 "Sentiment Analysis for Low-resource African Languages using Twitter Dataset". Sentiment analysis is one of the most widely studied applications in natural language processing. However, most prior work still focuses on a small number of high-resource languages. Building reliable sentiment analysis systems for low-resource languages remains challenging, due to the limited training data in this task. In this work, we propose to leverage language-adaptive and task-adaptive pretraining on African texts and study transfer learning with source language selection on top of an African language-centric pretrained language model. Our key findings are: (1) Adapting the pretrained model to the target language and task using a small yet relevant corpus improves performance remarkably by more than 10 F1 score points. (2) Selecting source languages with positive transfer gains during training can avoid harmful interference from dissimilar languages, leading to better results in multilingual and cross-lingual settings. In the shared task, our system wins 8 out of 15 tracks and, in particular, performs best in the multilingual evaluation.