mlsanityData to 14 Oct 2026

LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation

arXiv 2103.06418 ↗First posted 11 Mar 2021cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

The multilingual pre-trained language models (e.g, mBERT, XLM and XLM-R) have shown impressive performance on cross-lingual natural language understanding tasks. However, these models are computationally intensive and difficult to be deployed on resource-restricted devices. In this paper, we propose a simple yet effective distillation method (LightMBERT) for transferring the cross-lingual generalization ability of the multilingual BERT to a small student model. The experiment results empirically demonstrate the efficiency and effectiveness of LightMBERT, which is significantly better than the baselines and performs comparable to the teacher mBERT.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 19 of its 22 references that are in the map. First posted on arXiv 11 Mar 2021.

Builds on

19 of 22 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  7. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  9. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  10. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  11. 2019How multilingual is Multilingual BERT?cited by 645
  12. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
7 more
  1. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  2. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  3. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153
  4. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 120
  5. 2020Multilingual Alignment of Contextual Word Representationscited by 114
  6. 2019Small and Practical BERT Models for Sequence Labelingcited by 40
  7. 2020XtremeDistil: Multi-stage Distillation for Massive Multilingual Modelscited by 38

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 4.

5 papers.

  1. 2021Distilling the Knowledge of Romanian BERTs Using Multiple Teacherscited by 13
  2. 2022Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Modelscited by 5
  3. 2022Evaluating Generalizability of Fine-Tuned Models for Fake News Detectioncited by 3
  4. 2022Compressing Cross-Lingual Multi-Task Models at Qualtricscited by 1
  5. 2022CAMeMBERT: Cascading Assistant-Mediated Multilingual BERTcited by 0