mlsanityData to 7 Oct 2026

ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Model

arXiv 2404.02534 ↗First posted 3 Apr 2024cs.CL cs.AIcited by 2

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

In recent years, the development of pre-trained language models (PLMs) has gained momentum, showcasing their capacity to transcend linguistic barriers and facilitate knowledge transfer across diverse languages. However, this progress has predominantly bypassed the inclusion of very-low resource languages, creating a notable void in the multilingual landscape. This paper addresses this gap by introducing four tailored PLMs specifically finetuned for Angolan languages, employing a Multilingual Adaptive Fine-tuning (MAFT) approach. In this paper, we survey the role of informed embedding initialization and synthetic data in enhancing the performance of MAFT models in downstream tasks. We improve baseline over SOTA AfroXLMR-base (developed through MAFT) and OFA (an effective embedding initialization) by 12.3 and 3.8 points respectively.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 21 of its 28 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

21 of 28 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  5. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  6. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  7. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  8. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  9. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  10. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  11. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  12. 2019Generalized Data Augmentation for Low-Resource Translationcited by 53
9 more
  1. 2022AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languagescited by 46
  2. 2022Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptationcited by 44
  3. 2023SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)cited by 41
  4. 2022SERENGETI: Massively Multilingual Language Models for Africacited by 40
  5. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  6. 2019Improving Pre-Trained Multilingual Models with Vocabulary Expansioncited by 27
  7. 2021AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languagescited by 24
  8. 2023Taxi1500: A Multilingual Dataset for Text Classification in 1500 Languagescited by 23
  9. 2023Crosslingual Transfer Learning for Low-Resource Languages Based on Multilingual Colexification Graphscited by 11

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  2. 2025Synthetic Voice Data for Automatic Speech Recognition in African Languagescited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages
  2. 2026Bootstrapping Embeddings for Low Resource Languages
  3. 2025AfriMTEB and AfriE5: Benchmarking and Adapting Text Embedding Models for African Languages
  4. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias
  5. 2023AfroBench: How Good are Large Language Models on African Languages?
  6. 2024SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languages
  7. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models
  8. 2025NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communities