mlsanityData to 7 Oct 2026

Knowledge Distillation of Russian Language Models with Reduction of Vocabulary

arXiv 2205.02340 ↗First posted 4 May 2022cs.CL cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Today, transformer language models serve as a core component for majority of natural language processing tasks. Industrial application of such models requires minimization of computation time and memory footprint. Knowledge distillation is one of approaches to address this goal. Existing methods in this field are mainly focused on reducing the number of layers or dimension of embeddings/hidden representations. Alternative option is to reduce the number of tokens in vocabulary and therefore the embeddings matrix of the student model. The main problem with vocabulary minimization is mismatch between input sequences and output class distributions of a teacher and a student models. As a result, it is impossible to directly apply KL-based knowledge distillation. We propose two simple yet effective alignment techniques to make knowledge distillation to the students with reduced vocabulary. Evaluation of distilled models on a number of common benchmarks for Russian such as Russian SuperGLUE, SberQuAD, RuSentiment, ParaPhaser, Collection-3 demonstrated that our techniques allow to achieve compression from 17× to 49×, while maintaining quality of 1.7× compressed student with the full-sized vocabulary, but reduced number of Transformer layers only. We make our code and distilled models available.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 4 May 2022.

Builds on

32 of 46 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
20 more
  1. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  2. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  3. 2019Contrastive Representation Distillationcited by 734
  4. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  5. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  6. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  7. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  8. 2013Do Deep Nets Really Need to be Deep?cited by 466
  9. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  10. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
  11. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  12. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  13. 2021M6: A Chinese Multimodal Pretrainercited by 91
  14. 2019Knowledge Distillation from Internal Representationscited by 87
  15. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  16. 2020RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmarkcited by 53
  17. 2019SberQuAD -- Russian Reading Comprehension Dataset: Description and Analysiscited by 38
  18. 2019No Training Required: Exploring Random Encoders for Sentence Classificationcited by 30
  19. 2019Tensorized Embedding Layers for Efficient Model Compressioncited by 27
  20. 2019Transformer to CNN: Label-scarce distillation for efficient text classificationcited by 8

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2; 2025: 2.

7 papers.

  1. 2023A Family of Pretrained Transformer Language Models for Russiancited by 27
  2. 2023Monolingual and Cross-Lingual Knowledge Transfer for Topic Classificationcited by 1
  3. 2023HAlf-MAsked Model for Named Entity Sentiment analysiscited by 1
  4. 2024Stolen Subwords: Importance of Vocabularies for Machine Translation Model Stealingcited by 0
  5. 2024Minimizing PLM-Based Few-Shot Intent Detectorscited by 0
  6. 2025Knowledge Distillation of Domain-adapted LLMs for Question-Answering in Telecomcited by 0
  7. 2025Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Modelscited by 0