mlsanityData to 14 Oct 2026

WUSPruner: Second-Moment-Optimal Transforms for Structured Pruning

arXiv 2610.15394 ↗First posted 11 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Structured pruning is typically formulated as identifying which model dimensions to remove, often within the model's original representation. However, the effect of coordinate deletion fundamentally depends on the basis in which pruning is performed. This raises a more fundamental question: how should the basis be chosen specifically for coordinate deletion? We introduce WUSPruner, a transform-based framework for structured pruning of large language models. We formulate basis selection as a local linear reconstruction problem that directly minimizes the reconstruction loss induced by pruning. Inspired by the second-moment transformation in WUSH, we consider a WUS transform constructed jointly from activation and weight statistics. We show that under positive-definite second moments, WUS transform minimizes the local reconstruction loss over all invertible changes of basis. In the resulting basis, the activation and weight second moments are simultaneously diagonalized and balanced, causing the deletion error to decompose into independent spectral contributions and yielding a principled criterion for coordinate selection. Guided by this result, we develop a scalable pruning method based on blockwise WUS transformations and compact input projections. Experiments on Llama and Qwen models from 8B to 72B parameters show consistent improvements over structured pruning baselines across language modeling, knowledge, commonsense reasoning, mathematics, and code generation benchmarks, while also delivering practical reductions in inference latency and memory usage.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 35 references that are in the map. First posted on arXiv 11 Oct 2026.

Builds on

31 of 35 references.

References · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2016Pointer Sentinel Mixture Modelscited by 3,096
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
19 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  5. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  6. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  7. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  8. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  9. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  10. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  11. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  12. 2024HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuningcited by 123
  13. 2023Fluctuation-based Adaptive Structured Pruning for Large Language Modelscited by 111
  14. 2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 85
  15. 2024Evaluating Language Models for Efficient Code Generationcited by 73
  16. 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 49
  17. 2025DarwinLM: Evolutionary Structured Pruning of Large Language Modelscited by 18
  18. 2025WUSH: Near-Optimal Adaptive Transforms for LLM Quantizationcited by 15
  19. 2026SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-trainingcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient
  2. 2025From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models
  3. 2024FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models
  4. 2025Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation
  5. 2025Wanda++: Pruning Large Language Models via Regional Gradients
  6. 2025Pruning Laws for Large Language Models
  7. 2026Importance-Guided Basis Selection for Low-Rank Decomposition of Large Language Models
  8. 2024Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization