mlsanityData to 7 Oct 2026

Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models

arXiv 2312.07046 ↗First posted 12 Dec 2023cs.LG cs.CLcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Due to the substantial scale of Large Language Models (LLMs), the direct application of conventional compression methodologies proves impractical. The computational demands associated with even minimal gradient updates present challenges, particularly on consumer-grade hardware. This paper introduces an innovative approach for the parametric and practical compression of LLMs based on reduced order modelling, which entails low-rank decomposition within the feature space and re-parameterization in the weight space. Notably, this compression technique operates in a layer-wise manner, obviating the need for a GPU device and enabling the compression of billion-scale models within stringent constraints of both memory and time. Our method represents a significant advancement in model compression by leveraging matrix decomposition, demonstrating superior efficacy compared to the prevailing state-of-the-art structured pruning method.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 7 of its 14 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

7 of 14 references.

References · 7

  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  5. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  6. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  7. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simple
  2. 2025FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compression
  3. 2024Basis Selection: Low-Rank Decomposition of Pretrained Large Language Models for Target Applications
  4. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit
  5. 2026From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression
  6. 2024Lillama: Large Language Models Compression via Low-Rank Feature Distillation
  7. 2026Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression
  8. 2024Compressing Large Language Models with Automated Sub-Network Search