The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM

Neural network pruning is a promising technique to mitigate the excessive computational and memory requirements of large language models (LLMs). Despite its promise, however, progress in this area has diminished, as conventional methods are seemingly unable to surpass moderate sparsity levels (50-60%) without severely degrading model accuracy. This work breaks through the current impasse, presenting a principled and effective method called textttElsa, which achieves extreme sparsity levels of up to 90% while retaining high model fidelity. This is done by identifying several limitations in current practice, all of which can be traced back to their reliance on a surrogate objective formulation. textttElsa tackles this issue directly and effectively via standard and well-established constrained optimization techniques based on ADMM. Our extensive experiments across a wide range of models and scales show that textttElsa achieves substantial improvements over existing methods; e.g., it achieves 7.8× less perplexity than the best existing method on LLaMA-2-7B at 90% sparsity. Moreover, we show that textttElsa remains stable even at extreme sparsity (e.g., 95%), yielding up to ×3.98 inference speedup and ×7.80 memory compression over its dense counterpart. We also present textttElsa-L, a quantized variant that scales to extremely large models (27B), and establish its theoretical convergence guarantees.These results highlight meaningful progress in advancing the frontier of LLM sparsity, while promising that significant opportunities for further advancement may remain in directions that have so far attracted limited exploration.