mlsanityData to 14 Oct 2026

KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation

arXiv 2109.06243 ↗First posted 13 Sep 2021cs.CL cs.AIcited by 20

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

The development of over-parameterized pre-trained language models has made a significant contribution toward the success of natural language processing. While over-parameterization of these models is the key to their generalization power, it makes them unsuitable for deployment on low-capacity devices. We push the limits of state-of-the-art Transformer-based pre-trained language model compression using Kronecker decomposition. We use this decomposition for compression of the embedding layer, all linear mappings in the multi-head attention, and the feed-forward network modules in the Transformer layer. We perform intermediate-layer knowledge distillation using the uncompressed model as the teacher to improve the performance of the compressed model. We present our KroneckerBERT, a compressed version of the BERT_BASE model obtained using this framework. We evaluate the performance of KroneckerBERT on well-known NLP benchmarks and show that for a high compression factor of 19 (5% of the size of the BERT_BASE model), our KroneckerBERT outperforms state-of-the-art compression methods on the GLUE. Our experiments indicate that the proposed model has promising out-of-distribution robustness and is superior to the state-of-the-art compression methods on SQuAD.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 27 of its 36 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

27 of 36 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  12. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
15 more
  1. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  2. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  3. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  4. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 294
  5. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 195
  6. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 113
  7. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  8. 2020LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compressioncited by 45
  9. 2019Fully Quantized Transformer for Machine Translationcited by 36
  10. 2020FastFormers: Highly Efficient Transformer Models for Natural Language Understandingcited by 28
  11. 2019Tensorized Embedding Layers for Efficient Model Compressioncited by 27
  12. 2015Compression of Fully-Connected Layer in Neural Network by Kronecker Productcited by 9
  13. 2019Extremely Small BERT Models from Mixed-Vocabulary Trainingcited by 8
  14. 2017Slim Embedding Layers for Recurrent Neural Language Modelscited by 5
  15. 2020Compressing Language Models using Doped Kronecker Productscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 8; 2023: 4; 2024: 4; 2025: 1; 2026: 1.

20 papers.

Sort:

By area:

  1. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  2. 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 133
  3. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
  4. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  5. 2022Parameter-efficient Model Adaptation for Vision Transformerscited by 81
  6. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  7. 2021Kronecker Decomposition for GPT Compressioncited by 28
  8. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  9. 2021RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillationcited by 19
  10. 2024On Efficient Variants of Segment Anything Model: A Surveycited by 16
  11. 2022MorphTE: Injecting Morphology in Tensorized Embeddingscited by 5
  12. 2024TQCompressor: improving tensor decomposition methods in neural networks via permutationscited by 5
8 more
  1. 2022Kronecker Decomposition for Knowledge Graph Embeddingscited by 2
  2. 2022Do we need Label Regularization to Fine-tune Pre-trained Language Models?cited by 2
  3. 2023Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Modelscited by 2
  4. 2024EPE-P: Evidence-based Parameter-efficient Prompting for Multimodal Learning with Missing Modalitiescited by 2
  5. 2024Practical token pruning for foundation models in few-shot conversational virtual assistant systemscited by 1
  6. 2025Bridging the Gap Between Multimodal Foundation Models and World Modelscited by 1
  7. 2026Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretabilitycited by 1
  8. 2022CAMeMBERT: Cascading Assistant-Mediated Multilingual BERTcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Improving Knowledge Distillation for BERT Models: Loss Functions, Mapping Methods, and Weight Tuning
  2. 2022Exploring Extreme Parameter Compression for Pre-trained Language Models
  3. 2021A Short Study on Compressing Decoder-Based Language Models
  4. 2023Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective
  5. 2021Prune Once for All: Sparse Pre-Trained Language Models
  6. 2024MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models
  7. 2026Multi-Aspect Knowledge Distillation for Language Model with Low-rank Factorization
  8. 2022Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks