# Fragile Knowledge, Robust Instruction-Following: The Width Pruning Dichotomy in Llama-3.2

arXiv: https://arxiv.org/abs/2512.22671 · HTML: https://mlsanity.com/paper/2512.22671/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 27 Dec 2025.

## Builds on (most cited first, 17 of 17)

- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications](https://mlsanity.com/paper/2402.05162/) (2024)
- [Compact Language Models via Pruning and Knowledge Distillation](https://mlsanity.com/paper/2407.14679/) (2024)
- [The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction](https://mlsanity.com/paper/2312.13558/) (2023)
- [Instruction-Following Pruning for Large Language Models](https://mlsanity.com/paper/2501.02086/) (2025)
- [NIRVANA: Structured Pruning Reimagined for Large Language Model Compression](https://mlsanity.com/paper/2509.14230/) (2025)
- [Can pruning make Large Language Models more efficient?](https://mlsanity.com/paper/2310.04573/) (2023)
- [Efficient LLMs with AMP: Attention Heads and MLP Pruning](https://mlsanity.com/paper/2504.21174/) (2025)
- [When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs](https://mlsanity.com/paper/2510.22228/) (2025)
- [How Many Parameters Does Your Task Really Need? Task Specific Pruning with LLM-Sieve](https://mlsanity.com/paper/2505.18350/) (2025)
- [Dependency-Aware Semi-Structured Sparsity of GLU Variants in Large Language Models](https://mlsanity.com/paper/2405.01943/) (2024)
- [Truth Neurons](https://mlsanity.com/paper/2505.12182/) (2025)
- [Pruning Weights but Not Truth: Safeguarding Truthfulness While Pruning LLMs](https://mlsanity.com/paper/2509.00096/) (2025)

## Built on by (most cited first, 2 of 2)

- [LLMs can Compress LLMs: Adaptive Pruning by Agents](https://mlsanity.com/paper/2601.09694/) (2026)
- [Fairness Pruning: Locating Demographic Bias in GLU-MLP Layers via Differential Activations](https://mlsanity.com/paper/2607.28319/) (2026)

## Abstract (arXiv)

Structured width pruning of GLU-MLP layers in Llama-3.2 models, guided by the Peak-to-Peak Magnitude (PPM) criterion, reveals a systematic dichotomy in how reducing the expansion ratio affects different model capabilities. While performance on tasks relying on parametric knowledge (e.g., MMLU, GSM8K) and perplexity metrics degrades predictably with decreasing expansion ratios, instruction-following capabilities improve at the 2.4x equilibrium ratio (IFEval: +4.8 points / +46% in Llama-3.2-1B and +3.7 points / +39% in Llama-3.2-3B), and multi-step reasoning remains robust (MUSR). This pattern, observed consistently across both evaluated model sizes, challenges the prevailing assumption in compression research that pruning induces uniform degradation. To investigate this, we evaluated seven expansion ratio configurations using comprehensive benchmark suites that assess factual knowledge, mathematical reasoning, language comprehension, instruction-following, and truthfulness. Our analysis identifies the expansion ratio as a critical architectural parameter that selectively reshapes the model's task performance profile, rather than merely serving as a compression metric.