mlsanityData to 14 Oct 2026

Structured Pruning for Diverse Best-of-N Reasoning Optimization

arXiv 2506.03978 ↗First posted 4 Jun 2025cs.CL cs.LGcited by 4

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Model pruning in transformer-based language models, traditionally viewed as a means of achieving computational savings, can enhance the model's reasoning capabilities. In this work, we uncover a surprising phenomenon: the selective pruning of certain attention heads leads to improvements in reasoning performance, particularly on challenging tasks. Motivated by this observation, we propose SPRINT, a novel contrastive learning framework that dynamically selects the optimal head and layer to prune during inference. By aligning question embeddings with head embeddings, SPRINT identifies those pruned-head configurations that result in more accurate reasoning. Extensive experiments demonstrate that our method significantly outperforms traditional best-of-N and random head selection strategies on the MATH500 and GSM8K datasets.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 24 of its 45 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

24 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning12 Dec
  2. 2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computation7 Oct
  3. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training29 Sep

Other references · 21

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  9. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  10. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 332
  11. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  12. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 284
9 more
  1. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  2. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  3. 2019Structured Pruning of Large Language Modelscited by 203
  4. 2024pyvene: A Library for Understanding and Improving PyTorch Models via Interventionscited by 54
  5. 2024The Super Weight in Large Language Modelscited by 52
  6. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 46
  7. 2025Mixture-of-Personas Language Models for Population Simulationcited by 20
  8. 2025Probe-Free Low-Rank Activation Interventioncited by 5
  9. 2025Distributional Surgery for Language Model Activationscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 17
  2. 2026VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Modelcited by 9
  3. 2025Reasoning Planning for Language Modelscited by 2
  4. 2026A Game Theoretic Free Energy Analysis of Higher Order Synergy in Attention Heads of Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Unlocking Structured Pruning for Reasoning Language Models via Attention-Preserving MLP Trimming
  2. 2026From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models
  3. 2026Pruning via Causal Attribution Preserves Reasoning Performance in Large Language Models
  4. 2025Think Before You Prune: Self-Reflective Structured Pruning for Reasoning Language Models
  5. 2025Think Before You Prune: Selective Self-Generated Calibration for Pruning Large Reasoning Models
  6. 2026Batch-wise Adaptive Pruning: Periodic Neuron Activation-Aware Weight Pruning for Language Reasoning Model
  7. 2026OS-Pruner: Pruning Chains-of-Thought of Reasoning Models via Optimal Stopping
  8. 2026On the Limits of Layer Pruning for Generative Reasoning in Large Language Models