mlsanityData to 14 Oct 2026

Fast Tensorization of Neural Networks via Slice-wise Feature Distillation

arXiv 2605.19842 ↗First posted 19 May 2026cs.LGcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

We propose a scalable tensorization framework for neural network compression based on slice-wise feature distillation. Unlike conventional tensor decomposition methods that rely on costly global finetuning, our approach decomposes the network into slices consisting of either individual layers or blocks (e.g., convolutional layers or MLPs), or small groups of consecutive layers, and tensorizes each slice independently to reproduce the intermediate representations of the original pretrained model. This modular strategy improves accuracy recovery, reduces data requirements, and enables efficient parallel optimization. Experiments on ResNet-34 show significant gains over conventional global tensorization, achieving near-lossless compression at moderate compression rates with faster optimization. Results on GPT-2 XL further demonstrate the scalability of the method and its applicability to large-scale models, particularly in distributed settings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 26 of its 45 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

26 of 45 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2020Knowledge Distillation: A Surveycited by 1,603
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  6. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  7. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  8. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  9. 2018Model compression via distillation and quantizationcited by 289
  10. 2015Tensorizing Neural Networkscited by 250
  11. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  12. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 203
14 more
  1. 2022Language model compression with weighted low-rank factorizationcited by 195
  2. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  3. 2019A Tensorized Transformer for Language Modelingcited by 86
  4. 2020Heterogeneous Knowledge Distillation using Information Flow Modelingcited by 73
  5. 2023Pruning vs Quantization: Which is Better?cited by 46
  6. 2017Learning Compact Recurrent Neural Networks with Block-Term Tensor Decompositioncited by 33
  7. 2024CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networkscited by 32
  8. 2017CP-decomposition with Tensor Power Method for Convolutional Neural Networks Compressioncited by 30
  9. 2021Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operatorscited by 29
  10. 2019Compressing deep neural networks by matrix product operatorscited by 28
  11. 2022High-Order Pooling for Graph Neural Networks with Tensor Decompositioncited by 24
  12. 2020EDropout: Energy-Based Dropout and Pruning of Deep Neural Networkscited by 20
  13. 2021Joint Matrix Decomposition for Deep Convolutional Neural Networks Compressioncited by 10
  14. 2025Tensorization is a powerful but underexplored tool for compression and interpretability of neural networkscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026EinSort: Sorting is All We Need for Tensorizing LLM
  2. 2025Knowledge Distillation: Enhancing Neural Network Compression with Integrated Gradients
  3. 2021Convolutional Neural Network Compression through Generalized Kronecker Product Decomposition
  4. 2024TQCompressor: improving tensor decomposition methods in neural networks via permutations
  5. 2021Compact CNN Structure Learning by Knowledge Distillation
  6. 2023Quantization Aware Factorization for Deep Neural Network Compression
  7. 2026Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression
  8. 2026Efficient training for compact compression models via sequential distillation