mlsanityData to 7 Oct 2026

Sequential Functional Structured Tucker Compression for Large Language Model Attentions

arXiv 2610.00717 ↗First posted 30 Sep 2026cs.CL cs.AI stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Post-training compression of LLM attention is often formulated as independent matrix approximation, ignoring both the shared structure among attention projections and the representation shift introduced by earlier compression. We propose FTC, a sequential structured compression framework that adapts the approximation to the current compressed model while jointly exploiting the native Q/K/V head structure under a fixed storage budget. The output projection is handled separately to account for the changed post-attention representation. FTC requires neither fine-tuning nor gradient-based recovery. Across seven decoder-only LLMs from 6B to 32B parameters, FTC achieves the lowest WikiText-2 perplexity among the compared methods at every tested keep ratio on five modern GQA models, with the largest gains under aggressive compression. The improvements transfer to downstream tasks and remain substantial at the 32B scale.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 25 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

21 of 25 references.

References · 21

  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  3. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  4. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  5. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  6. 2021Accelerating Sparse Deep Neural Networkscited by 213
  7. 2022Language model compression with weighted low-rank factorizationcited by 192
  8. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
  9. 2025SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compressioncited by 63
  10. 2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 34
  11. 2023TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decompositioncited by 15
  12. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 15
9 more
  1. 2025AdaSVD: Adaptive Singular Value Decomposition for Large Language Modelscited by 13
  2. 2024LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Modelscited by 12
  3. 2025TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMscited by 11
  4. 2025COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximationcited by 5
  5. 2026AA-SVD : Anchored and Adaptive SVD for Large Language Model Compressioncited by 5
  6. 2025Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Modelscited by 4
  7. 2025Large Language Model Compression with Global Rank and Sparsity Optimizationcited by 4
  8. 2026Rethinking the Role of Tensor Decompositions in Post-Training LLM Compressioncited by 3
  9. 2025Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language Modelscited by 2

Built on by

Not cited yet.