mlsanityData to 14 Oct 2026

Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM

arXiv 2510.05544 ↗First posted 7 Oct 2025cs.CL cs.LGcited by 3

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Large language models (LLM) and vision-language models (VLM) have achieved state-of-the-art performance, but they impose significant memory and computing challenges in deployment. We present a novel low-rank compression framework to address this challenge. First, we upper bound the change of network loss via layer-wise activation-based compression errors, filling a theoretical gap in the literature. We then formulate low-rank model compression as a bi-objective optimization and prove that a single uniform tolerance yields surrogate Pareto-optimal heterogeneous ranks. Based on our theoretical insights, we propose Pareto-Guided Singular Value Decomposition (PGSVD), a zero-shot pipeline that improves activation-aware compression via Pareto-guided rank selection and alternating least-squares implementation. We apply PGSVD to both LLM and VLM, showing better accuracy at the same compression levels and inference speedup.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 47 references that are in the map. First posted on arXiv 7 Oct 2025.

Builds on

32 of 47 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
20 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  4. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  6. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  7. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  8. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  10. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  11. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  12. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  13. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  14. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  15. 2022Language model compression with weighted low-rank factorizationcited by 195
  16. 2025SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compressioncited by 67
  17. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 54
  18. 2022SPDY: Accurate Pruning with Speedup Guaranteescited by 36
  19. 2024Pushing the Limits of Large Language Model Quantization via the Linearity Theoremcited by 21
  20. 2024EvoPress: Accurate Dynamic Model Compression via Evolutionary Searchcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026AA-SVD : Anchored and Adaptive SVD for Large Language Model Compressioncited by 7
  2. 2026UMPIRE-Net: Unrolled Magnitude-Phase Regularization Network for Accelerated MRIcited by 2
  3. 2025INTERLACE: Interleaved Layer Pruning and Efficient Adaptation in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
  2. 2026Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression
  3. 2026IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression
  4. 2024Basis Selection: Low-Rank Decomposition of Pretrained Large Language Models for Target Applications
  5. 2024Characterizing the Accuracy -- Efficiency Trade-off of Low-rank Decomposition in Language Models
  6. 2026GA-SVD: Generation-aware Singular Value Decomposition for LLM Compression
  7. 2024Compressing Large Language Models using Low Rank and Low Precision Decomposition
  8. 2026Compressed-Sensing-Guided, Inference-Aware Structured Reduction for Large Language Models