mlsanityData to 14 Oct 2026

Contextual Text Embeddings for Twi

arXiv 2103.15963 ↗First posted 29 Mar 2021cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Transformer-based language models have been changing the modern Natural Language Processing (NLP) landscape for high-resource languages such as English, Chinese, Russian, etc. However, this technology does not yet exist for any Ghanaian language. In this paper, we introduce the first of such models for Twi or Akan, the most widely spoken Ghanaian language. The specific contribution of this research work is the development of several pretrained transformer language models for the Akuapem and Asante dialects of Twi, paving the way for advances in application areas such as Named Entity Recognition (NER), Neural Machine Translation (NMT), Sentiment Analysis (SA) and Part-of-Speech (POS) tagging. Specifically, we introduce four different flavours of ABENA -- A BERT model Now in Akan that is fine-tuned on a set of Akan corpora, and BAKO - BERT with Akan Knowledge only, which is trained from scratch. We open-source the model through the Hugging Face model hub and demonstrate its use via a simple sentiment classification example.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 23 of its 53 references that are in the map. First posted on arXiv 29 Mar 2021.

Builds on

23 of 53 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  8. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  9. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  10. 2015An Introduction to Convolutional Neural Networkscited by 514
  11. 2016Exploring the Limits of Language Modelingcited by 173
  12. 2014word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding methodcited by 112
11 more
  1. 2017Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Featurescited by 104
  2. 2017Transfer Learning across Low-Resource, Related Languages for Neural Machine Translationcited by 58
  3. 2020Masakhane -- Machine Translation For Africacited by 49
  4. 2016An Empirical Evaluation of doc2vec with Practical Insights into Document Embedding Generationcited by 48
  5. 2018Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spacescited by 34
  6. 2019Massive vs. Curated Word Embeddings for Low-Resourced Languages. The Case of Yor`ub'a and Twicited by 33
  7. 2018An Analysis of Neural Language Modeling at Multiple Scalescited by 31
  8. 2016An Ensemble Method to Produce High-Quality Word Embeddings (2016)cited by 13
  9. 2016A Neural Knowledge Language Modelcited by 13
  10. 2020Large-scale Transfer Learning for Low-resource Spoken Language Understandingcited by 11
  11. 2018Word2Vec and Doc2Vec in Unsupervised Sentiment Analysis of Clinical Discharge Summariescited by 3

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022Toward More Meaningful Resources for Lower-resourced Languagescited by 14