mlsanityData to 14 Oct 2026

Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming

arXiv 2610.19121 ↗First posted 13 Oct 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Structured pruning reduces LLM inference costs by removing model components to produce smaller dense models. However, existing methods can cause reasoning performance to collapse on long-form reasoning tasks such as AIME. We propose textitLayerTrim, a training-free structured pruning method that identifies redundant decoder layers and selectively trims their MLP features while preserving attention and decoder depth. LayerTrim is the first training-free structured pruning method designed to preserve long-form reasoning without gradients or recovery training. Across three reasoning language models and five benchmarks, LayerTrim outperforms existing structured pruning methods in most evaluated settings, improving average accuracy by up to 21.4 percentage points over RESP. On textttQwen3-14B, LayerTrim achieves 1.24× faster prefill and 19.7% lower peak GPU memory than the dense model, while better preserving reasoning ability. It also achieves the shortest average generation length among evaluated pruned models, while retaining the highest average accuracy. These results demonstrate that LayerTrim improves reasoning efficiency through both structural compression and reduced token generation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 16 of its 26 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

16 of 26 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026OBC-Prune: Outcome-Based Calibration for Large Reasoning Model Pruning15 Sep
  2. 2026Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning27 Apr
  3. 2026From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models26 Jan
  4. 2025Think Before You Prune: Self-Reflective Structured Pruning for Reasoning Language Models1 Dec
  5. 2025Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction15 Sep

Other references · 11

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  6. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  7. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 126
  8. 2025From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Modelscited by 6
  9. 2024GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compressioncited by 5
  10. 2026Locality-Aware Redundancy Pruning for LLM Depth Compressioncited by 2
  11. 2026CoCurve: Cross-Module Co-Pruning Curvature for Structured LLM Pruningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Limits of Layer Pruning for Generative Reasoning in Large Language Models
  2. 2025Structured Pruning for Diverse Best-of-N Reasoning Optimization
  3. 2026Revisiting the Effectiveness of LLM Pruning for Test-Time Scaling
  4. 2026Gradually Compacting Large Language Models for Reasoning Like a Boiling Frog
  5. 2025When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs
  6. 2023Just CHOP: Embarrassingly Simple LLM Compression
  7. 2025Can Pruning Improve Reasoning? Revisiting Long-CoT Compression with Capability in Mind for Better Reasoning
  8. 2025Think Before You Prune: Selective Self-Generated Calibration for Pruning Large Reasoning Models