# EvoPress: Accurate Dynamic Model Compression via Evolutionary Search

arXiv: https://arxiv.org/abs/2410.14649 · HTML: https://mlsanity.com/paper/2410.14649/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 54 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Builds on (most cited first, 30 of 31)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction](https://mlsanity.com/paper/2102.05426/) (2021)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [Extreme Compression of Large Language Models via Additive Quantization](https://mlsanity.com/paper/2401.06118/) (2024)
- [Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity](https://mlsanity.com/paper/2310.05175/) (2023)
- [Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks](https://mlsanity.com/paper/2102.08124/) (2021)
- [SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks](https://mlsanity.com/paper/2402.09025/) (2024)
- [NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search](https://mlsanity.com/paper/2105.14444/) (2021)
- [ZipLM: Inference-Aware Structured Pruning of Language Models](https://mlsanity.com/paper/2302.04089/) (2023)
- [BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation](https://mlsanity.com/paper/2402.16880/) (2024)
- [SPDY: Accurate Pruning with Speedup Guarantees](https://mlsanity.com/paper/2201.13096/) (2022)
- [N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning](https://mlsanity.com/paper/1709.06030/) (2017)
- [AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models](https://mlsanity.com/paper/2107.13686/) (2021)
- [Weight subcloning: direct initialization of transformers using larger pretrained ones](https://mlsanity.com/paper/2312.09299/) (2023)
- [Self-Adjusting Mutation Rates with Provably Optimal Success Rules](https://mlsanity.com/paper/1902.02588/) (2019)
- [Comma Selection Outperforms Plus Selection on OneMax with Randomly Planted Optima](https://mlsanity.com/paper/2304.09712/) (2023)
- [Self-Adjusting Population Sizes for Non-Elitist Evolutionary Algorithms: Why Success Rates Matter](https://mlsanity.com/paper/2104.05624/) (2021)
- [Self-adjusting Population Sizes for the (1, λ)-EA on Monotone Functions](https://mlsanity.com/paper/2204.00531/) (2022)

## Built on by (most cited first, 20 of 20)

- [2SSP: A Two-Stage Framework for Structured Pruning of LLMs](https://mlsanity.com/paper/2501.17771/) (2025)
- [TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models](https://mlsanity.com/paper/2504.09897/) (2025)
- [Statistically-Lossless Quantization of Large Language Models](https://mlsanity.com/paper/2605.02404/) (2026)
- [T'yr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization](https://mlsanity.com/paper/2503.09657/) (2025)
- [Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM](https://mlsanity.com/paper/2510.05544/) (2025)
- [Model Compression with Exact Budget Constraints via Riemannian Manifolds](https://mlsanity.com/paper/2605.00649/) (2026)
- [From Pruning to Grafting: Dynamic Knowledge Redistribution via Learnable Layer Fusion](https://mlsanity.com/paper/2411.14507/) (2024)
- [One-for-All Pruning: A Universal Model for Customized Compression of Large Language Models](https://mlsanity.com/paper/2505.12216/) (2025)
- [Think Before You Prune: Selective Self-Generated Calibration for Pruning Large Reasoning Models](https://mlsanity.com/paper/2511.18864/) (2025)
- [Rethinking Layer Redundancy: Calibration Matters More Than Search in LLM Depth Pruning](https://mlsanity.com/paper/2604.24938/) (2026)
- [Optimal Formats for Weight Quantisation](https://mlsanity.com/paper/2505.12988/) (2025)
- [TraceNAS: Zero-shot LLM Pruning via Gradient Trace Correlation](https://mlsanity.com/paper/2602.02891/) (2026)
- [WiSparse: Boosting LLM Inference Efficiency with Weight-Aware Mixed Activation Sparsity](https://mlsanity.com/paper/2602.14452/) (2026)
- [No Free Swap: Protocol-Dependent Layer Redundancy in Transformers](https://mlsanity.com/paper/2605.16234/) (2026)
- [Beyond Layer Importance in Layer-wise Sparsity: An Inter-Layer Perturbation-Absorption Perspective](https://mlsanity.com/paper/2606.15161/) (2026)
- [Prune, Update and Trim: Robust Structured Pruning for Large Language Models](https://mlsanity.com/paper/2605.18331/) (2026)
- [Agentic evolution of physically constrained foundation models](https://mlsanity.com/paper/2606.25532/) (2026)
- [Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization](https://mlsanity.com/paper/2607.22583/) (2026)
- [Correlation-Aware Structured Pruning for Large Language Models](https://mlsanity.com/paper/2609.22131/) (2026)
- [Output-aware Residual Stream Pruning for Large Language Models](https://mlsanity.com/paper/2609.35579/) (2026)

## Abstract (arXiv)

The high computational costs of large language models (LLMs) have led to a flurry of research on LLM compression, via methods such as quantization, sparsification, or structured pruning. A new frontier in this area is given by dynamic, non-uniform compression methods, which adjust the compression levels (e.g., sparsity) per-block or even per-layer in order to minimize accuracy loss, while guaranteeing a global compression threshold. Yet, current methods rely on estimating the importance of a given layer, implicitly assuming that layers contribute independently to the overall compression error. We begin from the motivating observation that this independence assumption does not generally hold for LLM compression: pruning a model further may even significantly recover performance. To address this, we propose EvoPress, a novel evolutionary framework for dynamic LLM compression. By formulating dynamic compression as a general optimization problem, EvoPress identifies optimal compression profiles in a highly efficient manner, and generalizes across diverse models and compression techniques. Via EvoPress, we achieve state-of-the-art performance for dynamic compression of Llama, Mistral, and Phi models, setting new benchmarks for structural pruning (block/layer dropping), unstructured sparsity, and quantization with dynamic bitwidths. Our code is available at https://github.com/IST-DASLab/EvoPress}.