mlsanityData to 14 Oct 2026

Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression

arXiv 2410.03765 ↗First posted 2 Oct 2024cs.CL cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have achieved remarkable breakthroughs. However, the huge number of parameters in LLMs require significant amount of memory storage in inference, which prevents their practical deployment in many applications. To reduce memory storage of LLMs, singular value decomposition (SVD) provides a promising solution to approximate weight matrices for compressing LLMs. In this paper, we take a step further to explore parameter sharing across different layers with SVD to achieve more effective compression for LLMs. Specifically, weight matrices in different layers are decomposed and represented as a linear combination of a set of shared basis vectors and unique coefficients. The types of weight matrices and the layer selection for basis sharing are examined when compressing LLMs to maintain the performance. Comprehensive experiments demonstrate that Basis Sharing outperforms state-of-the-art SVD-based compression approaches and parameter sharing techniques, especially under large compression ratios. Code is available at: https://github.com/TUDa-HWAI/Basis_Sharing

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 38 of its 45 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

38 of 45 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2023Mistral 7Bcited by 3,164
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
26 more
  1. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  3. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  4. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  5. 2018Universal Transformerscited by 647
  6. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  7. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  8. 2022Teaching Small Language Models to Reasoncited by 284
  9. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  10. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  11. 2022Language model compression with weighted low-rank factorizationcited by 195
  12. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  13. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  14. 2021Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformerscited by 43
  15. 2022In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Modelscited by 37
  16. 2019Careful Selection of Knowledge to solve Open Book Question Answeringcited by 35
  17. 2019Sharing Attention Weights for Fast Transformercited by 34
  18. 2023Lion: Adversarial Distillation of Proprietary Large Language Modelscited by 34
  19. 2021Leveraging redundancy in attention with Reuse Transformerscited by 33
  20. 2024EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Modelscited by 13
  21. 2024Dynamic Layer Tying for Parameter-Efficient Transformerscited by 10
  22. 2022Class-based Quantization for Neural Networkscited by 9
  23. 2023Class-Aware Pruning for Efficient Neural Networkscited by 9
  24. 2023PowerPruning: Selecting Weights and Activations for Power-Efficient Neural Network Accelerationcited by 8
  25. 2023Early-Exit with Class Exclusion for Efficient Inference of Neural Networkscited by 8
  26. 2023OplixNet: Towards Area-Efficient Optical Split-Complex Networks with Real-to-Complex Data Assignment and Knowledge Distillationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 20.

35 papers.

Sort:

By area:

  1. 2025Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinkingcited by 57
  2. 2026KV Packet: Recomputation-Free Context-Independent KV Caching for LLMscited by 10
  3. 2025CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generationcited by 9
  4. 2025CommonKV: Compressing KV Cache with Cross-layer Parameter Sharingcited by 9
  5. 2025Generalized Fisher-Weighted SVD: Scalable Kronecker-Factored Fisher Approximation for Compressing Large Language Modelscited by 8
  6. 2025Layer-wise dynamic rank for compressing large language modelscited by 8
  7. 2025ERC-SVD: Error-Controlled SVD for Large Language Model Compressioncited by 7
  8. 2026Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Densitycited by 7
  9. 2025CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learningcited by 6
  10. 2025LESA: Learnable LLM Layer Scaling-Upcited by 5
  11. 2025Share Your Attention: Transformer Weight Sharing via Matrix-based Dictionary Learningcited by 5
  12. 2024Learning Fine-grained Parameter Sharing via Sparse Tensor Decompositioncited by 3
23 more
  1. 2026A Practical Tensor-Network Compression Pipeline for Production-Scale Large Language Modelscited by 3
  2. 2026Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compressioncited by 3
  3. 2026FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fastcited by 3
  4. 2026Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compressioncited by 2
  5. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inferencecited by 2
  6. 2026Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMscited by 2
  7. 2025Neuronal Group Communication for Efficient Neural representationcited by 1
  8. 2025Rethinking Parameter Sharing as Graph Coloring for Structured Compressioncited by 1
  9. 2025SkipCat: Rank-Maximized Low-Rank Compression of Large Language Models via Shared Projection and Block Skippingcited by 1
  10. 2026Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compressioncited by 1
  11. 2025KernelDNA: Dynamic Kernel Sharing via Decoupled Naive Adapterscited by 0
  12. 2025Beyond Uniform SVD:Dual-Level Optimization across Columns and Modules for LLM Compressioncited by 0
  13. 2026Decompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compressioncited by 0
  14. 2026OptINC: Optical In-Network-Computing for Scalable Distributed Learningcited by 0
  15. 2026Why Geometric Continuity Emerges in Deep Neural Networks: Residual Connections and Rotational Symmetry Breakingcited by 0
  16. 2026Cross-Layer Subspace Coupling for LLM Compression: A Unifying Framework and Its Empirical Limitscited by 0
  17. 2026LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compressioncited by 0
  18. 2026Linear Reusable Neural Bases Architecture for Network Compressioncited by 0
  19. 2026DiffLUT-Net: Differentiable Training of FPGA LUT Networks with Learnable Connectivitycited by 0
  20. 2026GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compressioncited by 0
  21. 2026Learning Functional Subspaces for Neural Network Compressioncited by 0
  22. 2026Sequential Functional Structured Tucker Compression for Large Language Model Attentionscited by 0
  23. 2026The Functional Structure of Post-Compression Recovery in Low-Rank LLMscited by 0