# Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming

arXiv: https://arxiv.org/abs/2610.19121 · HTML: https://mlsanity.com/paper/2610.19121/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 16 of 16)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [LaCo: Large Language Model Pruning via Layer Collapse](https://mlsanity.com/paper/2402.11187/) (2024)
- [From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models](https://mlsanity.com/paper/2601.18091/) (2026)
- [From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models](https://mlsanity.com/paper/2510.18030/) (2025)
- [Think Before You Prune: Self-Reflective Structured Pruning for Reasoning Language Models](https://mlsanity.com/paper/2512.02185/) (2025)
- [GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compression](https://mlsanity.com/paper/2501.00339/) (2024)
- [Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction](https://mlsanity.com/paper/2509.12464/) (2025)
- [Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning](https://mlsanity.com/paper/2604.24938/) (2026)
- [Locality-Aware Redundancy Pruning for LLM Depth Compression](https://mlsanity.com/paper/2605.27786/) (2026)
- [CoCurve: Cross-Module Co-Pruning Curvature for Structured LLM Pruning](https://mlsanity.com/paper/2607.17568/) (2026)
- [OBC-Prune: Outcome-Based Calibration for Large Reasoning Model Pruning](https://mlsanity.com/paper/2609.17890/) (2026)

## Abstract (arXiv)

Structured pruning reduces LLM inference costs by removing model components to produce smaller dense models. However, existing methods can cause reasoning performance to collapse on long-form reasoning tasks such as AIME. We propose textitLayerTrim, a training-free structured pruning method that identifies redundant decoder layers and selectively trims their MLP features while preserving attention and decoder depth. LayerTrim is the first training-free structured pruning method designed to preserve long-form reasoning without gradients or recovery training. Across three reasoning language models and five benchmarks, LayerTrim outperforms existing structured pruning methods in most evaluated settings, improving average accuracy by up to 21.4 percentage points over RESP. On textttQwen3-14B, LayerTrim achieves 1.24× faster prefill and 19.7% lower peak GPU memory than the dense model, while better preserving reasoning ability. It also achieves the shortest average generation length among evaluated pruned models, while retaining the highest average accuracy. These results demonstrate that LayerTrim improves reasoning efficiency through both structural compression and reduced token generation.