mlsanityData to 7 Oct 2026

Vision Transformer Pruning Via Matrix Decomposition

arXiv 2308.10839 ↗First posted 21 Aug 2023cs.CV stat.COcited by 0

Not on the map. Why?

Abstract (arXiv)

This is a further development of Vision Transformer Pruning via matrix decomposition. The purpose of the Vision Transformer Pruning is to prune the dimension of the linear projection of the dataset by learning their associated importance score in order to reduce the storage, run-time memory, and computational demands. In this paper we further reduce dimension and complexity of the linear projection by implementing and comparing several matrix decomposition methods while preserving the generated important features. We end up selected the Singular Value Decomposition as the method to achieve our goal by comparing the original accuracy scores in the original Github repository and the accuracy scores of using those matrix decomposition methods, including Singular Value Decomposition, four versions of QR Decomposition, and LU factorization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 21 Aug 2023.

Builds on

24 of 36 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  6. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  7. 2017Decoupled Weight Decay Regularizationcited by 1,722
  8. 2020Pre-Trained Image Processing Transformercited by 891
  9. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  10. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  11. 2021Transformer in Transformercited by 632
  12. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
12 more
  1. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  2. 2015Deep Learning with Limited Numerical Precisioncited by 368
  3. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  4. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  5. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 115
  6. 2015Convolutional neural networks with low-rank regularizationcited by 102
  7. 2019Star-Transformercited by 94
  8. 2020SCOP: Scientific Control for Reliable Neural Network Pruningcited by 92
  9. 2021ISTR: End-to-End Instance Segmentation with Transformerscited by 52
  10. 2020Searching for Low-Bit Weights in Quantized Neural Networkscited by 43
  11. 2019Learning Low-Rank Approximation for CNNscited by 7
  12. 2023Topological Interpretations of GPT-3cited by 2

Built on by

Not cited yet.