mlsanityData to 14 Oct 2026

LatentLLM: Attention-Aware Joint Tensor Compression

arXiv 2505.18413 ↗First posted 23 May 2025cs.LG cs.AI cs.CLcited by 7

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Modern foundation models such as large language models (LLMs) and large multi-modal models (LMMs) require a massive amount of computational and memory resources. We propose a new framework to convert such LLMs/LMMs into a reduced-dimension latent structure. Our method extends a local activation-aware tensor decomposition to a global attention-aware joint tensor de-composition. Our framework can significantly improve the model accuracy over the existing model compression methods when reducing the latent dimension to realize computationally/memory-efficient LLMs/LLMs. We show the benefit on several benchmark including multi-modal reasoning tasks.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 48 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

39 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Q-VLM: Post-training Quantization for Large Vision-Language Models10 Oct

Other references · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  10. 2022Emergent Abilities of Large Language Modelscited by 2,512
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
26 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  3. 2024Mixtral of Expertscited by 1,823
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  6. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  8. 2019Green AIcited by 627
  9. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  10. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  11. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  12. 2023A Survey on Model Compression for Large Language Modelscited by 269
  13. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  14. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  15. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  16. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  17. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
  18. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compressioncited by 72
  19. 2024Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Modelscited by 67
  20. 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 59
  21. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  22. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 44
  23. 2022Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscited by 43
  24. 2024SparseLLM: Towards Global Pruning for Pre-trained Language Modelscited by 23
  25. 2024Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrixcited by 20
  26. 2024PC-LoRA: Low-Rank Adaptation for Progressive Model Compression with Knowledge Distillationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMscited by 6
  2. 2025Globally optimized SVD compression of LLMs via Fermi-function-based rank selection and gauge fixingcited by 3
  3. 2026Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMscited by 2
  4. 2025SkipCat: Rank-Maximized Low-Rank Compression of Large Language Models via Shared Projection and Block Skippingcited by 1
  5. 2026TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Flycited by 1
  6. 2026Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretabilitycited by 1
  7. 2026LaMoC: Loss-Aware Modular Compression for LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs
  2. 2026Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression
  3. 2023Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models
  4. 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference
  5. 2023Quantization-Aware and Tensor-Compressed Training of Transformers for Natural Language Understanding
  6. 2025Tensor Product Attention Is All You Need
  7. 2025FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compression
  8. 2024Compressing Large Language Models with Automated Sub-Network Search