mlsanityData to 7 Oct 2026

Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices

arXiv 2102.07071 ↗First posted 14 Feb 2021cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Structured matrices, such as those derived from Kronecker products (KP), are effective at compressing neural networks, but can lead to unacceptable accuracy loss when applied to large models. In this paper, we propose the notion of doping -- addition of an extremely sparse matrix to a structured matrix. Doping facilitates additional degrees of freedom for a small number of parameters, allowing them to independently diverge from the fixed structure. To train LSTMs with doped structured matrices, we introduce the additional parameter matrix while slowly annealing its sparsity level. However, we find that performance degrades as we slowly sparsify the doping matrix, due to co-matrix adaptation (CMA) between the structured and the sparse matrices. We address this over dependence on the sparse matrix using a co-matrix dropout regularization (CMR) scheme. We provide empirical evidence to show that doping, CMA and CMR are concepts generally applicable to multiple structured matrices (Kronecker Product, LMF, Hybrid Matrix Decomposition). Additionally, results with doped kronecker product matrices demonstrate state-of-the-art accuracy at large compression factors (10 - 25x) across 4 natural language processing applications with minor loss in accuracy. Doped KP compression technique outperforms previous state-of-the art compression results by achieving 1.3 - 2.4x higher compression factor at a similar accuracy, while also beating strong alternatives like pruning and low-rank methods by a large margin (8% or more). Additionally, we show that doped KP can be deployed on commodity hardware using the current software stack and achieve 2.5 - 5.5x inference run-time speed-up over baseline.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 51 of its 75 references that are in the map. First posted on arXiv 14 Feb 2021.

Builds on

51 of 75 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  4. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  5. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  6. 2019The State of Sparsity in Deep Neural Networkscited by 471
  7. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  8. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  9. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
  10. 2014Recurrent Neural Network Regularizationcited by 189
  11. 2016Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Fillingcited by 137
  12. 2016Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modelingcited by 87
38 more
  1. 2016Quasi-Recurrent Neural Networkscited by 87
  2. 2017Simple Recurrent Units for Highly Parallelizable Recurrencecited by 60
  3. 2017CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-CirculantWeight Matricescited by 53
  4. 2019SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollerscited by 52
  5. 2019Splitting Steepest Descent for Growing Neural Architecturescited by 52
  6. 2017Skip RNN: Learning to Skip State Updates in Recurrent Neural Networkscited by 49
  7. 2015Structured Transforms for Small-Footprint Deep Learningcited by 41
  8. 2020TinyLSTMs: Efficient Neural Speech Enhancement for Hearing Aidscited by 41
  9. 2017Structured Bayesian Pruning via Log-Normal Multiplicative Noisecited by 39
  10. 2019FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Networkcited by 38
  11. 2016Recurrent Highway Networkscited by 36
  12. 2017Compressing Recurrent Neural Network with Tensor Traincited by 28
  13. 2018Online Embedding Compression for Text Classification using Low Rank Matrix Factorizationcited by 22
  14. 2018Learning Compressed Transforms with Low Displacement Rankcited by 19
  15. 2017Learning to Skim Textcited by 18
  16. 2017Learning Intrinsic Sparse Structures within Long Short-Term Memorycited by 18
  17. 2020Systolic Tensor Array: An Efficient Structured-Sparse GEMM Accelerator for Mobile CNN Inferencecited by 17
  18. 2017Neural Speed Reading via Skim-RNNcited by 14
  19. 2017Factorization tricks for LSTM networkscited by 13
  20. 2020PERMDNN: Efficient Compressed DNN Architecture with Permuted Diagonal Matricescited by 12
  21. 2019Compressing RNNs for IoT devices by 15-38x using Kronecker Productscited by 11
  22. 2019FixyNN: Efficient Hardware for Mobile Computer Vision via Transfer Learningcited by 10
  23. 2019DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modelingcited by 10
  24. 2015Exploiting Local Structures with the Kronecker Layer in Convolutional Networkscited by 7
  25. 2019Compression of Recurrent Neural Networks for Efficient Language Modelingcited by 7
  26. 2019Structured Pruning of Recurrent Neural Networks through Neuron Selectioncited by 7
  27. 2020Rank and run-time aware compression of NLP Applicationscited by 5
  28. 2017Neural Networks Compression for Language Modelingcited by 4
  29. 2019Ternary Hybrid Neural-Tree Networks for Highly Constrained IoT Applicationscited by 4
  30. 2020Compressing Language Models using Doped Kronecker Productscited by 4
  31. 2018Mobile Machine Learning Hardware at ARM: A Systems-on-Chip (SoC) Perspectivecited by 3
  32. 2018Structured Weight Matrices-Based Hardware Accelerators in Deep Neural Networks: FPGAs and ASICscited by 3
  33. 2019Ternary MobileNets via Per-Layer Hybrid Filter Bankscited by 3
  34. 2018Retraining-Based Iterative Weight Quantization for Deep Neural Networkscited by 2
  35. 2018Pyramidal Recurrent Unit for Language Modelingcited by 2
  36. 2019Pushing the limits of RNN Compressioncited by 2
  37. 2018DeepTwist: Learning Model Compression via Occasional Weight Distortioncited by 1
  38. 2019Measuring scheduling efficiency of RNNs for NLP applicationscited by 1

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 1; 2024: 1; 2025: 0; 2026: 1.

5 papers.

  1. 2022Machine Learning for Microcontroller-Class Hardware: A Reviewcited by 28
  2. 2021S2TA: Exploiting Structured Sparsity for Energy-Efficient Mobile CNN Accelerationcited by 19
  3. 2024Enabling Unstructured Sparse Acceleration on Structured Sparse Acceleratorscited by 8
  4. 2023Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiencycited by 7
  5. 2026Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deploymentcited by 0