mlsanityData to 14 Oct 2026

ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations

arXiv 2506.02818 ↗First posted 3 Jun 2025cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) demonstrate impressive results in natural language processing tasks but require a significant amount of computational and memory resources. Structured matrix representations are a promising way for reducing the number of parameters of these models. However, it seems unrealistic to expect that weight matrices of pretrained models can be accurately represented by structured matrices without any fine-tuning. To overcome this issue, we utilize the fact that LLM output is invariant under certain orthogonal transformations of weight matrices. This insight can be leveraged to identify transformations that significantly improve the compressibility of weights within structured classes. The proposed approach is applicable to various types of structured matrices that support efficient projection operations. Code is available at https://github.com/GrishKate/ProcrustesGPT

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 21 of its 41 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

21 of 41 references.

References · 21

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  8. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  9. 2022Language model compression with weighted low-rank factorizationcited by 195
  10. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 123
  11. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 106
  12. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
9 more
  1. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 97
  2. 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 59
  3. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 48
  4. 2018GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinkingcited by 32
  5. 2021Kronecker Decomposition for GPT Compressioncited by 28
  6. 2018Online Embedding Compression for Text Classification using Low Rank Matrix Factorizationcited by 22
  7. 2024Group and Shuffle: Efficient Structured Orthogonal Parametrizationcited by 11
  8. 2024Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimizationcited by 10
  9. 2024TQCompressor: improving tensor decomposition methods in neural networks via permutationscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2026A Hamiltonian-Inspired Local-Operator Ansatz for Slimming Large Language Modelscited by 0
  3. 2026Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretrainingcited by 0