Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2
Not on the map. Why?
Abstract (arXiv)
Structured width pruning of GLU-MLP layers in Llama-3.2 models, guided by the Peak-to-Peak Magnitude (PPM) criterion, reveals a systematic dichotomy in how reducing the expansion ratio affects different model capabilities. While performance on tasks relying on parametric knowledge (e.g., MMLU, GSM8K) and perplexity metrics degrades predictably with decreasing expansion ratios, instruction-following capabilities improve at the 2.4x equilibrium ratio (IFEval: +4.8 points / +46% in Llama-3.2-1B and +3.7 points / +39% in Llama-3.2-3B), and multi-step reasoning remains robust (MUSR). This pattern, observed consistently across both evaluated model sizes, challenges the prevailing assumption in compression research that pruning induces uniform degradation. To investigate this, we evaluated seven expansion ratio configurations using comprehensive benchmark suites that assess factual knowledge, mathematical reasoning, language comprehension, instruction-following, and truthfulness. Our analysis identifies the expansion ratio as a critical architectural parameter that selectively reshapes the model's task performance profile, rather than merely serving as a compression metric.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 17 of its 21 references that are in the map. First posted on arXiv 27 Dec 2025.
Builds on
17 of 21 references.
References · 17
- 2023A Survey of Large Language Modelscited by 2,576
- 2020GLU Variants Improve Transformercited by 1,804
- 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
- 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
- 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
- 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
- 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
- 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 123
- 2025Instruction-Following Pruning for Large Language Modelscited by 17
- 2025NIRVANA: Structured Pruning Reimagined for Large Language Model Compressioncited by 6
- 2023Can pruning make Large Language Models more efficient?cited by 5
- 2025Efficient LLMs with AMP: Attention Heads and MLP Pruningcited by 5
5 more
- 2025When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMscited by 5
- 2025How Many Parameters Does Your Task Really Need? Task Specific Pruning with LLM-Sievecited by 3
- 2024Dependency-Aware Semi-Structured Sparsity of GLU Variants in Large Language Modelscited by 2
- 2025Truth Neuronscited by 2
- 2025Pruning Weights but Not Truth: Safeguarding Truthfulness While Pruning LLMscited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling
- 2025Can Pruning Improve Reasoning? Revisiting Long-CoT Compression with Capability in Mind for Better Reasoning
- 2026From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models
- 2026Do Instruction-Tuned Models Always Perform Better Than Base Models? Evidence from Math and Domain-Shifted Benchmarks
- 2023Is Bigger and Deeper Always Better? Probing LLaMA Across Scales and Layers
- 2024Does Instruction Tuning Make LLMs More Consistent?
- 2026Weight Pruning Amplifies Bias: A Multi-Method Study of Compressed LLMs for Edge AI
- 2025How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients