mlsanityData to 14 Oct 2026

BLAST: Block-Level Adaptive Structured Matrices for Efficient Deep Neural Network Inference

arXiv 2410.21262 ↗First posted 28 Oct 2024cs.LG cs.AI stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

Large-scale foundation models have demonstrated exceptional performance in language and vision tasks. However, the numerous dense matrix-vector operations involved in these large networks pose significant computational challenges during inference. To address these challenges, we introduce the Block-Level Adaptive STructured (BLAST) matrix, designed to learn and leverage efficient structures prevalent in the weight matrices of linear layers within deep learning models. Compared to existing structured matrices, the BLAST matrix offers substantial flexibility, as it can represent various types of structures that are either learned from data or computed from pre-existing weight matrices. We demonstrate the efficiency of using the BLAST matrix for compressing both language and vision tasks, showing that (i) for medium-sized models such as ViT and GPT-2, training with BLAST weights boosts performance while reducing complexity by 70% and 40%, respectively; and (ii) for large foundation models such as Llama-7B and DiT-XL, the BLAST matrix achieves a 2x compression while exhibiting the lowest performance degradation among all tested structured matrices. Our code is available at https://github.com/changwoolee/BLAST.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 48 of its 68 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

48 of 68 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2023GPT-4 Technical Reportcited by 18,396
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  10. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  11. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  12. 2023Segment Anythingcited by 8,234
36 more
  1. 2022Scalable Diffusion Models with Transformerscited by 6,054
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2016Improved Techniques for Training GANscited by 2,970
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  13. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  14. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  15. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  16. 2014Convex Optimization: Algorithms and Complexitycited by 575
  17. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
  18. 2021Generating Images with Sparse Representationscited by 266
  19. 2021Accelerating Sparse Deep Neural Networkscited by 223
  20. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  21. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 106
  22. 2015Convolutional neural networks with low-rank regularizationcited by 103
  23. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  24. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 78
  25. 2019Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizationscited by 76
  26. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 76
  27. 2020Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descentcited by 71
  28. 2024Adapting to Unknown Low-Dimensional Structures in Score-Based Diffusion Modelscited by 56
  29. 2024Breaking the Curse of Dimensionality: Diffusion Models Efficiently Learn Low-Dimensional Distributionscited by 53
  30. 2021Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorizationcited by 45
  31. 2023The Power of Preconditioning in Overparameterized Low-Rank Matrix Sensingcited by 33
  32. 2023Understanding Deep Representation Learning via Layerwise Feature Compression and Discriminationcited by 32
  33. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  34. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 20
  35. 2023Differentiable Learning of Generalized Structured Matrices for Efficient Deep Neural Networkscited by 5
  36. 2024Data-free Weight Compress and Denoise for Large Language Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Modelscited by 8
  2. 2025Memory-Efficient Acceleration of Block Low-Rank Foundation Models on Resource Constrained GPUscited by 1
  3. 2026DyRA: Dynamic Residual Approximation for Efficient Matrix Multiplication in DNNscited by 0