mlsanityData to 14 Oct 2026

Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression

arXiv 2606.03465 ↗First posted 2 Jun 2026cs.LG cs.AIcited by 4

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Post-training compression is essential for deploying large language models (LLMs) under tight resource constraints. Tensor decompositions have emerged as a promising direction, offering compact parameterizations well suited to Transformer weight structures. However, existing studies evaluate these methods in narrow settings, leaving unclear whether tensorization is effective at large-scale deployment. We systematically evaluate tensor compression across dense and MoE architectures, establishing performance trade-offs grounded in both empirical analysis and theoretical analysis. We identify a fundamental mismatch between the shared subspaces assumed by tensor decompositions and the heterogeneous representations learned by modern LLMs, thereby delineating their practical limits and clarifying their viable role in large-scale deployment. The code is available at https://github.com/brain-lab-research/TT-LLM.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 36 of its 50 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

36 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs26 Jan
  2. 2024The Unreasonable Ineffectiveness of the Deeper Layers26 Mar
  3. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction21 Dec

Other references · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  8. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  9. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  10. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  11. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  12. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
21 more
  1. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  2. 2024Massive Activations in Large Language Modelscited by 294
  3. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  4. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  5. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  6. 2024The Super Weight in Large Language Modelscited by 52
  7. 2025Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectivescited by 48
  8. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 27
  9. 2024AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuningcited by 25
  10. 2025Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsitycited by 18
  11. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 17
  12. 2023GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Modelcited by 15
  13. 2024Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMscited by 14
  14. 2025MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMscited by 9
  15. 2024TRAWL: Tensor Reduced and Approximated Weights for Large Language Modelscited by 5
  16. 2025FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compressioncited by 5
  17. 2025COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximationcited by 5
  18. 2026HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantizationcited by 5
  19. 2026SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compressioncited by 4
  20. 2025HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMscited by 3
  21. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimizationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devicescited by 2
  2. 2026Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretabilitycited by 1
  3. 2026Pre-registered tests of solid-state-physics-inspired LLM compression: a cluster-level negative result at small-language-model scalecited by 0
  4. 2026Sequential Functional Structured Tucker Compression for Large Language Model Attentionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Scaling Laws for Post Training Quantized Large Language Models
  2. 2025LatentLLM: Attention-Aware Joint Tensor Compression
  3. 2026From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression
  4. 2023The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models
  5. 2023Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models
  6. 2026Break Through the Compression Bottleneck: From Theory to Practice
  7. 2024Basis Selection: Low-Rank Decomposition of Pretrained Large Language Models for Target Applications
  8. 2024Communication Compression for Tensor Parallel LLM Inference