mlsanityData to 7 Oct 2026

GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compression

arXiv 2501.00339 ↗First posted 31 Dec 2024cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Recent studies have demonstrated that many layers are functionally redundant in large language models (LLMs), enabling model compression by removing these layers to reduce inference cost. While such approaches can improve efficiency, indiscriminate layer pruning often results in significant performance degradation. In this paper, we propose GRASP (Gradient-based Retention of Adaptive Singular Parameters), a novel compression framework that mitigates this issue by preserving sensitivity-aware singular values. Unlike direct layer pruning, GRASP leverages gradient-based attribution on a small calibration dataset to adaptively identify and retain critical singular components. By replacing redundant layers with only a minimal set of parameters, GRASP achieves efficient compression while maintaining strong performance with minimal overhead. Experiments across multiple LLMs show that GRASP consistently outperforms existing compression methods, achieving 90% of the original model's performance under 20% compression ratio.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 31 Dec 2024.

Builds on

32 of 46 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
20 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  5. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  6. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  7. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  8. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  9. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  10. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  11. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
  12. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  13. 2022Language model compression with weighted low-rank factorizationcited by 192
  14. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  15. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 102
  16. 2024SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blockscited by 96
  17. 2019Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobiancited by 67
  18. 2019ChID: A Large-scale Chinese IDiom Dataset for Cloze Testcited by 65
  19. 2024Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimizationcited by 9
  20. 2024Surgical Feature-Space Decomposition of LLMs: Why, When and How?cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Astra: Activation-Space Tail-Eigenvector Low-Rank Adaptation of Large Language Modelscited by 0
  2. 2026SHIFT-LLM: Distribution Shift Correction in Depth-Pruned LLMscited by 0
  3. 2026Importance-Aware Low-Rank Distillation of Diffusion Transformerscited by 0