mlsanityData to 14 Oct 2026

Activation- and Influence-Aware Ranks (AIR): Function-Preserving SVD Compression for LLMs

arXiv 2606.19993 ↗First posted 18 Jun 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We present Activation- and Influence-Aware Ranks (AIR), an SVD-based LLM compression framework that guides each weight matrix's low-rank approximation with a backward-signal influence metric. Starting from the activation-aware optimum of SVD-LLM(W), AIR runs a single closed-form alternating least squares (ALS) sweep that integrates influence element-wise under a monotone-descent guarantee. AIR is layer-local and composes orthogonally with end-to-end methods: alone it exceeds ACIP, and AIR+LoRA outperforms it further. AIR improves perplexity over SVD-LLM(W) by >18% at <=60% parameter retention, matches its quality with ~90% less calibration data, and turns parameter savings into FLOP, peak-memory, and per-token latency gains.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

29 of 44 references.

References · 29

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
17 more
  1. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  2. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  3. 2014New insights and perspectives on the natural gradient methodcited by 443
  4. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  5. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  6. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
  7. 2022Language model compression with weighted low-rank factorizationcited by 195
  8. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  9. 2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 120
  10. 2019Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruningcited by 89
  11. 2025SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compressioncited by 67
  12. 2021ECQx: Explainability-Driven Quantization for Low-Bit and Sparse DNNscited by 15
  13. 2024Pruning By Explaining Revisited: Optimizing Attribution Methods to Prune CNNs and Transformerscited by 11
  14. 2025An Overview of Low-Rank Structures in the Training and Adaptation of Large Modelscited by 9
  15. 2025Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Modelscited by 8
  16. 2024Evaluating the Impact of Compression Techniques on Task-Specific Performance of Large Language Modelscited by 3
  17. 2025Choose Your Model Size: Any Compression of Large Language Models Without Re-Computationcited by 3

Built on by

Not cited yet.