mlsanityData to 13 Oct 2026

CAMeMBERT: Cascading Assistant-Mediated Multilingual BERT

arXiv 2212.11456 ↗First posted 22 Dec 2022cs.CLcited by 0

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Large language models having hundreds of millions, and even billions, of parameters have performed extremely well on a variety of natural language processing (NLP) tasks. Their widespread use and adoption, however, is hindered by the lack of availability and portability of sufficiently large computational resources. This paper proposes a knowledge distillation (KD) technique building on the work of LightMBERT, a student model of multilingual BERT (mBERT). By repeatedly distilling mBERT through increasingly compressed toplayer distilled teacher assistant networks, CAMeMBERT aims to improve upon the time and space complexities of mBERT while keeping loss of accuracy beneath an acceptable threshold. At present, CAMeMBERT has an average accuracy of around 60.1%, which is subject to change after future improvements to the hyperparameters used in fine-tuning.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 10 of its 12 references that are in the map. First posted on arXiv 22 Dec 2022.

Builds on

10 of 12 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation13 Sep

Other references · 9

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,281
  4. 2020Knowledge Distillation: A Surveycited by 1,590
  5. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 914
  6. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 741
  7. 2019Improved Knowledge Distillation via Teacher Assistantcited by 569
  8. 2021Annealing Knowledge Distillationcited by 61
  9. 2021LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillationcited by 5

Built on by

Not cited yet.