mlsanityData to 7 Oct 2026

Compressing LLMs with MoP: Mixture of Pruners

arXiv 2602.06127 ↗First posted 5 Feb 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The high computational demands of Large Language Models (LLMs) motivate methods that reduce parameter count and accelerate inference. In response, model pruning emerges as an effective strategy, yet current methods typically focus on a single dimension-depth or width. We introduce MoP (Mixture of Pruners), an iterative framework that unifies these dimensions. At each iteration, MoP generates two branches-pruning in depth versus pruning in width-and selects a candidate to advance the path. On LLaMA-2 and LLaMA-3, MoP advances the frontier of structured pruning, exceeding the accuracy of competing methods across a broad set of compression regimes. It also consistently outperforms depth-only and width-only pruning. Furthermore, MoP translates structural pruning into real speedup, reducing end-to-end latency by 39% at 40% compression. Finally, extending MoP to the vision-language model LLaVA-1.5, we notably improve computational efficiency and demonstrate that text-only recovery fine-tuning can restore performance even on visual tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 48 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

38 of 48 references.

References · 38

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
26 more
  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  2. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  3. 2019Green AIcited by 620
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  5. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  6. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  7. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  8. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  9. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  10. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  11. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 112
  12. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  13. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
  14. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  15. 2023The LLM Surgeoncited by 36
  16. 2024Streamlining Redundant Layers to Compress Large Language Modelscited by 32
  17. 2024DISP-LLM: Dimension-Independent Structural Pruning for Large Language Modelscited by 24
  18. 2024LOVA3: Learning to Visual Question Answering, Asking and Assessmentcited by 21
  19. 2025SlimLLM: Accurate Structured Pruning for Large Language Modelscited by 18
  20. 2025You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learningcited by 16
  21. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  22. 2024AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deploymentcited by 12
  23. 2025Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenationcited by 6
  24. 2025Efficient LLMs with AMP: Attention Heads and MLP Pruningcited by 5
  25. 2025A Simple Linear Patch Revives Layer-Pruned Large Language Modelscited by 4
  26. 2025ModHiFi: Identifying High Fidelity predictive components for Model Modificationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric
  2. 2025Pruning Laws for Large Language Models
  3. 2026Structural Pruning of Large Vision Language Models: A Comprehensive Study on Pruning Dynamics, Recovery, and Data Efficiency
  4. 2023Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models
  5. 2025SPAP: Structured Pruning via Alternating Optimization and Penalty Methods
  6. 2023Just CHOP: Embarrassingly Simple LLM Compression
  7. 2025MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures
  8. 2025Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study