mlsanityData to 7 Oct 2026

Krony-PT: GPT2 compressed with Kronecker Products

arXiv 2412.12351 ↗First posted 16 Dec 2024cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce Krony-PT, a compression technique for GPT-2 based on Kronecker products. We specifically target the feed-forward weights of each transformer block, and systematically compress the feed-forward layer matrices to various degrees. We introduce a modified Van Loan decomposition to initialize new Kronecker factors, and also propose a new pruning-based initialization technique. Our method compresses the original 124M-parameter GPT-2 to various smaller models, ranging from 80M to 96M. Our 81M model variant outperforms DistilGPT2 on next-token prediction across all standard language modeling datasets, and shows competitive or comparable performance with significantly larger Kronecker-based compressions of GPT-2.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 33 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

17 of 33 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2020Knowledge Distillation: A Surveycited by 1,578
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  10. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  11. 2022Language model compression with weighted low-rank factorizationcited by 192
  12. 2020A Statistical Framework for Low-bitwidth Training of Deep Neural Networkscited by 51
5 more
  1. 2020Towards Fully 8-bit Integer Inference for the Transformer Modelcited by 35
  2. 2021Kronecker Decomposition for GPT Compressioncited by 27
  3. 2015Compression of Fully-Connected Layer in Neural Network by Kronecker Productcited by 9
  4. 2020Kronecker Attention Networkscited by 9
  5. 2024TQCompressor: improving tensor decomposition methods in neural networks via permutationscited by 5

Built on by

Not cited yet.