mlsanityData to 7 Oct 2026

NoWag: A Unified Framework for Shape Preserving Compression of Large Language Models

arXiv 2504.14569 ↗First posted 20 Apr 2025cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) exhibit remarkable performance across various natural language processing tasks but suffer from immense computational and memory demands, limiting their deployment in resource-constrained environments. To address this challenge, we propose NoWag (Normalized Weight and Activation Guided Compression), a unified framework for one-shot shape preserving compression algorithms. We apply NoWag to compress Llama-2 (7B, 13B, 70B) and Llama-3 (8B, 70B) models using two popular shape-preserving techniques: vector quantization (NoWag-VQ) and unstructured/semi-structured pruning (NoWag-P). Our results show that NoWag-VQ significantly outperforms state-of-the-art one-shot vector quantization methods, while NoWag-P performs competitively against leading pruning techniques. These findings highlight underlying commonalities between these compression paradigms and suggest promising directions for future research. Our code is available at https://github.com/LawrenceRLiu/NoWag

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 35 of its 58 references that are in the map. First posted on arXiv 20 Apr 2025.

Builds on

35 of 58 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
23 more
  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  3. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  5. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  6. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  7. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  8. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  9. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  10. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  11. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  12. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
  13. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 86
  14. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 78
  15. 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 63
  16. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 59
  17. 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 58
  18. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 53
  19. 2023Intriguing Properties of Quantization at Scalecited by 48
  20. 2023Pruning vs Quantization: Which is Better?cited by 45
  21. 2023Can Unstructured Pruning Reduce the Depth in Deep Neural Networks?cited by 20
  22. 2024Pruning Large Language Models with Semi-Structural Adaptive Sparse Trainingcited by 20
  23. 2025ClusComp: A Simple Paradigm for Model Compression and Efficient Finetuningcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorizationcited by 2