mlsanityData to 14 Oct 2026

DiBA: Diagonal and Binary Matrix Approximation for Neural Network Weight Compression

arXiv 2605.05994 ↗First posted 7 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose DiBA (Diagonal and Binary Matrix Approximation), a compact matrix factorization for neural network weight compression. Many components of modern networks, including linear layers, 1×1 convolutions, attention projections, and embedding layers, have dense matrix weights. DiBA approximates A∈ℝm× n by widehat A=D1B1D2B2D3, where D1,D2,D3 are diagonal matrices and B1,B2 are 0/1 binary matrices. The intermediate dimension k controls the trade-off between theoretical storage and approximation accuracy. For matrix-vector products, DiBA decomposes dense multiplication into three element-wise scaling operations and two binary mixing operations, reducing the floating-point multiplication count from mn to m+k+n. For optimization, we introduce DiBA-Greedy, an alternating solver that combines closed-form least-squares updates for the diagonal factors with exact one-bit improvement tests for the binary factors. We also introduce DiBARD (DiBA with Retuning only Diagonal factors), which replaces dense-matrix layers by DiBA factors, freezes the binary matrices, and retunes only the diagonal entries on downstream data. This preserves compact binary mixing without discrete search during adaptation. On 40 dense weight matrices extracted from public pretrained models, DiBA-Greedy yields consistent SNR improvements as the theoretical storage ratio increases. After DiBA replacement in two component-replacement studies, DiBARD improves DistilBERT/WikiText masked-token accuracy from 0.4447 to 0.5210 and Speech Commands test accuracy for an Audio Spectrogram Transformer from 0.7684 to 0.9781 without reoptimizing the binary factors.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 50 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

42 of 50 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  9. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  11. 2019Searching for MobileNetV3cited by 1,994
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
30 more
  1. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  2. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  4. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  5. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  6. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  7. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  8. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  9. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  10. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  11. 2019Learned Step Size Quantizationcited by 628
  12. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  13. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  14. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  15. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  16. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  17. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  18. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  19. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  20. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
  21. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 261
  22. 2015Tensorizing Neural Networkscited by 250
  23. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 217
  24. 2016Trained Ternary Quantizationcited by 208
  25. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 195
  26. 2015Compressing Neural Networks with the Hashing Trickcited by 150
  27. 2019Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networkscited by 148
  28. 2017Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMMcited by 61
  29. 2018ProxQuant: Quantized Neural Networks via Proximal Operatorscited by 59
  30. 2018Loss-aware Weight Quantization of Deep Networkscited by 31

Built on by

Not cited yet.