mlsanityData to 14 Oct 2026

When Compression Scores Cannot Decide: Information Boundaries for Group-Robust LLM Pruning

arXiv 2608.02940 ↗First posted 3 Aug 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

A stable compression score can still select the worse model. In our dense study, a split-half reliable path-quadratic score predicted a 16.1% gain, while the selected endpoints were 6.0--7.7% worse than two controls. We ask what a compression statistic can justify when deployment cares about the worst supplied group. We treat each statistic as an information interface. Its observation leaves a fiber of compatible endpoint-risk tables, and only orders fixed across that fiber are identified. Cone and fiber identities quantify the remaining uncertainty, while matched observations reverse endpoint order for pooled moments, group-local moments, and reference-path curvature. Sequential composition adds one state variable: the slack from each group risk to the current maximum. This vector determines every unrestricted one-step response, and a margin condition keeps the active group fixed along paths with bounded relative drift. The experiments follow the same ladder. Across three dense LLMs, an early-preserving allocation reduces worst-group perplexity inflation by 12.6--20.9%; target-matched complete-menu selection improves over its references by 2.7--8.0%. Across all 16 routed layers of OLMoE, pooled endpoint refresh lowers held-out worst-group teacher KL by 15.8% over the best static score. A compute-matched hard-max trajectory ends 32.7% worse than pooled, and neither adaptive trajectory improves excess NLL. Local evidence can narrow a menu. Complete endpoints rank that menu, while multistep claims also require control of the evolving active face and future candidates.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 37 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

27 of 37 references.

References · 27

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  5. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  6. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  7. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  8. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 165
  9. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  10. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  11. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 57
  12. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 53
15 more
  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  2. 2024BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocationcited by 45
  3. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compressioncited by 44
  4. 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observationscited by 20
  5. 2025Why Do Some Inputs Break Low-Bit LLM Quantization?cited by 9
  6. 2024The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Informationcited by 8
  7. 2024On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruningcited by 6
  8. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  9. 2021Optimal Decision Rules Under Partial Identificationcited by 3
  10. 2024Are Compressed Language Models Less Subgroup Robust?cited by 3
  11. 2026SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMscited by 3
  12. 2026Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Codingcited by 3
  13. 2025M-Wanda: Improving One-Shot Pruning for Multilingual LLMscited by 2
  14. 2026Hilbert Operator for Progressive Encoding (HOPE): A Mathematical Framework for Deconstructing Learned Representations in Deep Networkscited by 2
  15. 2025PreMoE: Proactive Inference for Efficient Mixture-of-Expertscited by 1

Built on by

Not cited yet.