mlsanityData to 14 Oct 2026

Matrix Compression via Randomized Low Rank and Low Precision Factorization

arXiv 2310.11028 ↗First posted 17 Oct 2023cs.LG cs.IT math.ITcited by 25

Not on the map. Why?

Abstract (arXiv)

Matrices are exceptionally useful in various fields of study as they provide a convenient framework to organize and manipulate data in a structured manner. However, modern matrices can involve billions of elements, making their storage and processing quite demanding in terms of computational resources and memory usage. Although prohibitively large, such matrices are often approximately low rank. We propose an algorithm that exploits this structure to obtain a low rank decomposition of any matrix A as A ≈ LR, where L and R are the low rank factors. The total number of elements in L and R can be significantly less than that in A. Furthermore, the entries of L and R are quantized to low precision formats -- compressing A by giving us a low rank and low precision factorization. Our algorithm first computes an approximate basis of the range space of A by randomly sketching its columns, followed by a quantization of the vectors constituting this basis. It then computes approximate projections of the columns of A onto this quantized basis. We derive upper bounds on the approximation error of our algorithm, and analyze the impact of target rank and quantization bit-budget. The tradeoff between compression ratio and approximation accuracy allows for flexibility in choosing these parameters based on specific application requirements. We empirically demonstrate the efficacy of our algorithm in image compression, nearest neighbor classification of image and text embeddings, and compressing the layers of LlaMa-7b. Our results illustrate that we can achieve compression ratios as aggressive as one bit per matrix coordinate, all while surpassing or maintaining the performance of traditional compression techniques.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 44 of its 88 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

44 of 88 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  6. 2019Searching for MobileNetV3cited by 1,994
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  10. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  11. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  12. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
32 more
  1. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  2. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  3. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
  4. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 233
  5. 2019Structured Pruning of Large Language Modelscited by 203
  6. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 160
  7. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  8. 2016Distributed Mean Estimation with Limited Communicationcited by 136
  9. 2018When and Why are Pre-trained Word Embeddings Useful for Neural Machine Translation?cited by 126
  10. 2015Convolutional neural networks with low-rank regularizationcited by 103
  11. 2020Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Networkcited by 74
  12. 2020Breaking the Communication-Privacy-Accuracy Trilemmacited by 68
  13. 2021EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learningcited by 55
  14. 2021DRIVE: One-bit Distributed Mean Estimationcited by 52
  15. 2021FedNL: Making Newton-Type Methods Applicable to Federated Learningcited by 52
  16. 2020Uncertainty Principle for Communication Compression in Distributed and Federated Learning and the Search for an Optimal Compressorcited by 45
  17. 2020LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compressioncited by 45
  18. 2021Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorizationcited by 45
  19. 2016Practical sketching algorithms for low-rank matrix approximationcited by 38
  20. 2019RATQ: A Universal Fixed-Length Quantizer for Stochastic Optimizationcited by 23
  21. 2020Transform Quantization for CNN (Convolutional Neural Network) Compressioncited by 20
  22. 2014Low-Rank Modeling and Its Applications in Image Analysiscited by 15
  23. 2019On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compressioncited by 14
  24. 2022Correlated quantization for distributed mean estimation and optimizationcited by 14
  25. 2020Precise expressions for random projections: Low-rank approximation and randomized Newtoncited by 13
  26. 2021General Low-rank Matrix Optimization: Geometric Analysis and Sharper Boundscited by 12
  27. 2016Lecture Notes on Randomized Linear Algebracited by 11
  28. 2021Fast and Accurate Randomized Algorithms for Low-rank Tensor Decompositionscited by 11
  29. 2021Communication-Efficient Distributed Optimization with Quantized Preconditionerscited by 8
  30. 2021Efficient Randomized Subspace Embeddings for Distributed Optimization under a Communication Budgetcited by 5
  31. 2021Unique sparse decomposition of low rank matricescited by 4
  32. 2022Minimax Optimal Quantization of Linear Models: Information-Theoretic Limits and Efficient Algorithmscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 7; 2026: 7.

25 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  2. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  3. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  4. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  5. 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 59
  6. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 25
  7. 2024Reassessing Layer Pruning in LLMs: New Insights and Methodscited by 24
  8. 2025Q-KVComm: Efficient Multi-Agent Communication Via Adaptive KV Cache Compressioncited by 9
  9. 2024DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantizationcited by 7
  10. 2026Model Merging in the Essential Subspacecited by 7
  11. 2026NSC-SL: A Bandwidth-Aware Neural Subspace Compression for Communication-Efficient Split Learningcited by 4
  12. 2025DReSS: Data-driven Regularized Structured Streamlining for Large Language Modelscited by 3
13 more
  1. 2024In-depth Analysis of Low-rank Matrix Factorisation in a Federated Settingcited by 2
  2. 2025Exploring Federated Pruning for Large Language Modelscited by 2
  3. 2025DFQ-ViT: Data-Free Quantization for Vision Transformers without Fine-tuningcited by 2
  4. 2024Getting Free Bits Back from Rotational Symmetries in LLMscited by 1
  5. 2025PIP: Perturbation-based Iterative Pruning for Large Language Modelscited by 1
  6. 2025Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decompositioncited by 1
  7. 2026SALAAD: Sparse And Low-Rank Adaptation via ADMM for Large Language Model Inferencecited by 1
  8. 2026Beyond Student: An Asymmetric Network for Neural Network Inheritancecited by 1
  9. 2026Data-Aware Model Merging in the Essential Subspace for Multi-Task Knowledge Aggregationcited by 1
  10. 2024FiRST: Finetuning Router-Selective Transformers for Input-Adaptive Latency Reductioncited by 0
  11. 2025Clustering-Based Low-Rank Matrix Approximation for Medical Image Compressioncited by 0
  12. 2026Stochastic Rounding Increases Small Singular Valuescited by 0
  13. 2026Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blockscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Optimal Quantization for Matrix Multiplication
  2. 2024Quantization-aware Matrix Factorization for Low Bit Rate Image Compression
  3. 2025HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs
  4. 2022Learning Low-Rank Representations for Model Compression
  5. 2026LoPRo: Enhancing Low-Rank Quantization via Permuted Block-Wise Rotation
  6. 2025A3 : an Analytical Low-Rank Approximation Framework for Attention
  7. 2023LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression
  8. 2024Compression for Better: A General and Stable Lossless Compression Framework