mlsanityData to 14 Oct 2026

Pruning Large Language Models via Accuracy Predictor

arXiv 2309.09507 ↗First posted 18 Sep 2023cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Large language models(LLMs) containing tens of billions of parameters (or even more) have demonstrated impressive capabilities in various NLP tasks. However, substantial model size poses challenges to training, inference, and deployment so that it is necessary to compress the model. At present, most model compression for LLMs requires manual design of pruning features, which has problems such as complex optimization pipeline and difficulty in retaining the capabilities of certain parts of the model.Therefore, we propose a novel pruning approach: firstly, a training set of a certain number of architecture-accuracy pairs is established, and then a non-neural model is trained as an accuracy predictor. Using the accuracy predictor to further optimize the search space and search, the optimal model can be automatically selected. Experiments show that our proposed approach is effective and efficient. Compared with the baseline, the perplexity(PPL) on Wikitext2 and PTB dropped by 9.48% and 5,76% respectively, and the average accuracy of MMLU increased by 6.28%.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

24 of 34 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  11. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  12. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
12 more
  1. 2019Importance Estimation for Neural Network Pruningcited by 541
  2. 2023Explainability for Large Language Models: A Surveycited by 333
  3. 2018Neural Architecture Optimizationcited by 167
  4. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  5. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  6. 2023Norm Tweaking: High-performance Low-bit Quantization of Large Language Modelscited by 33
  7. 2023DisWOT: Student Architecture Search for Distillation WithOut Trainingcited by 29
  8. 2023EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantizationcited by 22
  9. 2022Prior-Guided One-shot Neural Architecture Searchcited by 12
  10. 2023GP-NAS-ensemble: a model for NAS Performance Predictioncited by 7
  11. 2023RD-NAS: Enhancing One-shot Supernet Ranking Ability via Ranking Distillation from Zero-cost Proxiescited by 7
  12. 2020Accuracy Prediction with Non-neural Model for Neural Architecture Searchcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3.

5 papers.

  1. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  2. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 62
  3. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 38
  4. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  5. 2024Adaptive Pruning for Large Language Models with Structural Importance Awarenesscited by 0