# Distilling the Knowledge of Romanian BERTs Using Multiple Teachers

arXiv: https://arxiv.org/abs/2112.12650 · HTML: https://mlsanity.com/paper/2112.12650/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 23 Dec 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT](https://mlsanity.com/paper/1909.05840/) (2019)
- [Empirical Analysis of Predictive Algorithms for Collaborative Filtering](https://mlsanity.com/paper/1301.7363/) (2013)
- [The birth of Romanian BERT](https://mlsanity.com/paper/2009.08712/) (2020)
- [One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers](https://mlsanity.com/paper/2106.01023/) (2021)
- [Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression](https://mlsanity.com/paper/2109.03228/) (2021)
- [Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecture](https://mlsanity.com/paper/2005.03454/) (2020)
- [MOROCO: The Moldavian and Romanian Dialectal Corpus](https://mlsanity.com/paper/1901.06543/) (2019)
- [Introducing RONEC -- the Romanian Named Entity Corpus](https://mlsanity.com/paper/1909.01247/) (2019)
- [SaRoCo: Detecting Satire in a Novel Romanian Corpus of News Articles](https://mlsanity.com/paper/2105.06456/) (2021)
- [Clustering Word Embeddings with Self-Organizing Maps. Application on LaRoSeDa -- A Large Romanian Sentiment Data Set](https://mlsanity.com/paper/2101.04197/) (2021)
- [PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptors](https://mlsanity.com/paper/2108.01139/) (2021)
- [LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation](https://mlsanity.com/paper/2103.06418/) (2021)

## Built on by (most cited first, 13 of 13)

- [The Zeno's Paradox of `Low-Resource' Languages](https://mlsanity.com/paper/2410.20817/) (2024)
- [HistNERo: Historical Named Entity Recognition for the Romanian Language](https://mlsanity.com/paper/2405.00155/) (2024)
- [Romanian Multiword Expression Detection Using Multilingual Adversarial Training and Lateral Inhibition](https://mlsanity.com/paper/2304.11350/) (2023)
- [Adversarial Capsule Networks for Romanian Satire Detection and Sentiment Analysis](https://mlsanity.com/paper/2306.07845/) (2023)
- ["Vorbecsti Romanecste?" A Recipe to Train Powerful Romanian LLMs with English Instructions](https://mlsanity.com/paper/2406.18266/) (2024)
- [RoMemes: A multimodal meme corpus for the Romanian language](https://mlsanity.com/paper/2410.15497/) (2024)
- [Explainability-Driven Leaf Disease Classification Using Adversarial Training and Knowledge Distillation](https://mlsanity.com/paper/2401.00334/) (2023)
- [Enhancing Romanian Offensive Language Detection through Knowledge Distillation, Multi-Task Learning, and Data Augmentation](https://mlsanity.com/paper/2409.20498/) (2024)
- [An Open-Domain QA System for e-Governance](https://mlsanity.com/paper/2206.08046/) (2022)
- [RoBERTweet: A BERT Language Model for Romanian Tweets](https://mlsanity.com/paper/2306.06598/) (2023)
- [UniBERT: Adversarial Training for Language-Universal Representations](https://mlsanity.com/paper/2503.12608/) (2025)
- [SaRoHead: Detecting Satire in a Multi-Domain Romanian News Headline Dataset](https://mlsanity.com/paper/2504.07612/) (2025)
- [RoLegalGEC: Legal Domain Grammatical Error Detection and Correction Dataset for Romanian](https://mlsanity.com/paper/2604.19593/) (2026)

## Abstract (arXiv)

Running large-scale pre-trained language models in computationally constrained environments remains a challenging problem yet to be addressed, while transfer learning from these models has become prevalent in Natural Language Processing tasks. Several solutions, including knowledge distillation, network quantization, or network pruning have been previously proposed; however, these approaches focus mostly on the English language, thus widening the gap when considering low-resource languages. In this work, we introduce three light and fast versions of distilled BERT models for the Romanian language: Distil-BERT-base-ro, Distil-RoBERT-base, and DistilMulti-BERT-base-ro. The first two models resulted from the individual distillation of knowledge from two base versions of Romanian BERTs available in literature, while the last one was obtained by distilling their ensemble. To our knowledge, this is the first attempt to create publicly available Romanian distilled BERT models, which were thoroughly evaluated on five tasks: part-of-speech tagging, named entity recognition, sentiment analysis, semantic textual similarity, and dialect identification. Our experimental results argue that the three distilled models offer performance comparable to their teachers, while being twice as fast on a GPU and ~35% smaller. In addition, we further test the similarity between the predictions of our students versus their teachers by measuring their label and probability loyalty, together with regression loyalty - a new metric introduced in this work.