mlsanityData to 7 Oct 2026

mALBERT: Is a Compact Multilingual BERT Model Still Worth It?

arXiv 2403.18338 ↗First posted 27 Mar 2024cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Within the current trend of Pretained Language Models (PLM), emerge more and more criticisms about the ethical andecological impact of such models. In this article, considering these critical remarks, we propose to focus on smallermodels, such as compact models like ALBERT, which are more ecologically virtuous than these PLM. However,PLMs enable huge breakthroughs in Natural Language Processing tasks, such as Spoken and Natural LanguageUnderstanding, classification, Question--Answering tasks. PLMs also have the advantage of being multilingual, and,as far as we know, a multilingual version of compact ALBERT models does not exist. Considering these facts, wepropose the free release of the first version of a multilingual compact ALBERT model, pre-trained using Wikipediadata, which complies with the ethical aspect of such a language model. We also evaluate the model against classicalmultilingual PLMs in classical NLP tasks. Finally, this paper proposes a rare study on the subword tokenizationimpact on language performances.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 22 of its 36 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

22 of 36 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
10 more
  1. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  2. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  3. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  4. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  5. 2020End-to-End Slot Alignment and Recognition for Cross-Lingual NLUcited by 82
  6. 2019UER: An Open-Source Toolkit for Pre-training Modelscited by 61
  7. 2017Linguistically Motivated Vocabulary Reduction for Neural Machine Translation from Turkish to Englishcited by 23
  8. 2018Morphological and Language-Agnostic Word Segmentation for NMTcited by 16
  9. 2022On the Usability of Transformers-based models for a French Question-Answering taskcited by 8
  10. 2022Benchmarking Transformers-based models on French Spoken Language Understanding taskscited by 3

Built on by

Not cited yet.