mlsanityData to 14 Oct 2026

Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning

arXiv 2310.06694 ↗First posted 10 Oct 2023cs.CL cs.AI cs.LGcited by 415

Not on the map. Why?

Cited by 39 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The popularity of LLaMA (Touvron et al., 2023a;b) and other recently emerged moderate-sized large language models (LLMs) highlights the potential of building smaller yet powerful LLMs. Regardless, the cost of training such models from scratch on trillions of tokens remains high. In this work, we study structured pruning as an effective means to develop smaller LLMs from pre-trained, larger models. Our approach employs two key techniques: (1) targeted structured pruning, which prunes a larger model to a specified target shape by removing layers, heads, and intermediate and hidden dimensions in an end-to-end manner, and (2) dynamic batch loading, which dynamically updates the composition of sampled data in each training batch based on varying losses across different domains. We demonstrate the efficacy of our approach by presenting the Sheared-LLaMA series, pruning the LLaMA2-7B model down to 1.3B and 2.7B parameters. Sheared-LLaMA models outperform state-of-the-art open-source models of equivalent sizes, such as Pythia, INCITE, OpenLLaMA and the concurrent TinyLlama models, on a wide range of downstream and instruction tuning evaluations, while requiring only 3% of compute compared to training such models from scratch. This work provides compelling evidence that leveraging existing LLMs with structured pruning is a far more cost-effective approach for building competitive small-scale LLMs

Key facts. Cited by 415 papers in the map (data through 14 Oct 2026). Builds on 62 of its 85 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

62 of 85 references.

References · 62

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
38 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  6. 2020GLU Variants Improve Transformercited by 1,804
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  11. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  13. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  14. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  15. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  16. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  17. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  18. 2023StarCoder: may the source be with you!cited by 799
  19. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  20. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  21. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  22. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  23. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  24. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  25. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  26. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  27. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  28. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  29. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  30. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  31. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  32. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  33. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  34. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  35. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  36. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  37. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  38. 2018A Simple Method for Commonsense Reasoningcited by 223

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 15; 2024: 171; 2025: 148; 2026: 81.

415 papers.

Sort:

By area:

  1. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  2. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  3. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  4. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  5. 2024Generative Representational Instruction Tuningcited by 233
  6. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  7. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  8. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  9. 2023Efficient Large Language Models: A Surveycited by 186
  10. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  11. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 157
  12. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
38 more
  1. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  2. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 132
  3. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
  4. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 126
  5. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
  6. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  7. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  8. 2024Universal Neurons in GPT2 Language Modelscited by 102
  9. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 96
  10. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  11. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  12. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 82
  13. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  14. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
  15. 2024Repetition Improves Language Model Embeddingscited by 72
  16. 2024CLLMs: Consistency Large Language Modelscited by 72
  17. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 67
  18. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 66
  19. 2024On the Societal Impact of Open Foundation Modelscited by 65
  20. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 65
  21. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  22. 2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 62
  23. 2024Finding Transformer Circuits with Edge Pruningcited by 60
  24. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 57
  25. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 56
  26. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  27. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  28. 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 50
  29. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 47
  30. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 46
  31. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 45
  32. 2024Flextron: Many-in-One Flexible Large Language Modelcited by 45
  33. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 44
  34. 2024SlimGPT: Layer-wise Structured Pruning for Large Language Modelscited by 44
  35. 2024Everybody Prune Now: Structured Pruning of LLMs with only Forward Passescited by 43
  36. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Modelscited by 42
  37. 2023Initializing Models with Larger Onescited by 39
  38. 2023LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recoverycited by 38

50 most cited of 415 All 415 (JSON)