# Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too

arXiv: https://arxiv.org/abs/2610.18516 · HTML: https://mlsanity.com/paper/2610.18516/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Builds on (most cited first, 27 of 27)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023)
- [SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2403.07378/) (2024)
- [Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch](https://mlsanity.com/paper/2102.04010/) (2021)
- [ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models](https://mlsanity.com/paper/2312.05821/) (2023)
- [Scaling Up Influence Functions](https://mlsanity.com/paper/2112.03052/) (2021)
- [What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions](https://mlsanity.com/paper/2405.13954/) (2024)
- [The LLM Surgeon](https://mlsanity.com/paper/2312.17244/) (2023)
- [Influence Functions for Scalable Data Attribution in Diffusion Models](https://mlsanity.com/paper/2410.13850/) (2024)
- [Model-Preserving Adaptive Rounding](https://mlsanity.com/paper/2505.22988/) (2025)
- [SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs](https://mlsanity.com/paper/2405.16325/) (2024)
- [OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition](https://mlsanity.com/paper/2409.13652/) (2024)
- [ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models](https://mlsanity.com/paper/2406.07831/) (2024)
- [Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuning](https://mlsanity.com/paper/2602.17835/) (2026)
- [3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs](https://mlsanity.com/paper/2603.01376/) (2026)

## Abstract (arXiv)

Post-training pruning enables efficient deployment, but conventional methods do not explicitly preserve internal-gradient alignment between sparse and dense models. We study novel connections between downstream performance of compressed models and the ability to retain gradient information via influence-function preservation. Central to this connection is a curvature-aware objective that accounts for both layer activations and backpropagated gradients. Despite its promise, this objective remains underexplored in pruning due to a lack of efficient solvers. We introduce textttCACTUS, a novel ADMM-based optimizer supporting both pruning and sparse-plus-low-rank decompositions of pre-trained weights. Compared to LLM-Surgeon which targets the same objective, textttCACTUS can achieve over an order-of-magnitude speedup. At 50% pruning of Qwen3-4B, textttCACTUS reduces the C4 perplexity gap to the dense model by 45%, improves compressed-to-dense per-sample gradient cosine similarity on WikiText-2 from 0.670 to 0.755, and runs over 28× faster than LLM-Surgeon on substantially weaker hardware. By preserving influence functions, textttCACTUS supports effective data selection. On Llama-3.2-3B, fine-tuning a fresh dense model on the 5% of data selected by textttCACTUS achieves 49.44% accuracy on BIG-Bench Hard, compared with 47.22% using ALPS and 49.54% using dense-model selection. Finally, integrating textttCACTUS with LoGRA, a recent gradient-projection strategy, on Qwen3-4B with MLP layers pruned to 2:4 sparsity and a rank-64 correction retains a 0.711 cosine alignment with dense LoGRA-projected C4 gradients.