Task-Aware Spectral Pruning: A Mixture-of-Masks Framework for Efficient LLM Inference

arXiv 2609.29499 ↗First posted 24 Aug 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Static pruning imposes one sparse structure on every prompt, even though reasoning, retrieval, generation, coding, and translation can depend on different parts of a language model. We introduce Task-Aware Spectral Pruning (TASP), a post-training framework that calibrates module-level spectral descriptors against measured task-specific ablation effects, closes grouped-query-attention and SwiGLU dependencies during sparse-mask construction, and routes each user turn to one compiled mask that remains fixed throughout prefill and decoding. A module-disjoint pilot first determines whether the spectral signal is informative before full calibration. Under the stated retrospective operating rule, the pilot passes on the evaluated Llama-3-8B and Llama-3-70B checkpoints but rejects Qwen2.5-1.5B, demonstrating that applicability is model-dependent rather than universal. At a 43% active-FLOP reduction, the Llama-3-70B benchmark harness retains 97.7 +/- 0.2% of the dense BF16 score. In the deployment-matched INT8-weight/BF16-compute runtime on a single A100 80GB, the compiled sparse path retains 97.3 +/- 0.2% relative to dense BF16 and reduces decode latency from 45.2 +/- 0.4 to 31.3 +/- 0.4 ms/token, yielding a 1.44x speedup. Factorized ablations, disjoint-module tests, compiled structured baselines, routing-corruption studies, and an explicit 136-GPU-hour calibration audit further delimit the source and operating regime of these gains

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 23 of its 40 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

23 of 40 references.

References · 23

2020Language Models are Few-Shot Learnerscited by 32,419
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2021Evaluating Large Language Models Trained on Codecited by 8,320
2020Measuring Massive Multitask Language Understandingcited by 7,963
2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
2021Program Synthesis with Large Language Modelscited by 3,637
2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631

11 more

2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
2017Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behaviorcited by 25
2025IG-Pruning: Input-Guided Block Pruning for Large Language Modelscited by 3
2026LLMs can Compress LLMs: Adaptive Pruning by Agentscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way