Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Models
Not on the map. Why?
Abstract (arXiv)
Matrix and tensor-guided parametrization for Natural Language Processing (NLP) models is fundamentally useful for the improvement of the model's systematic efficiency. However, the internal links between these two algebra structures and language model parametrization are poorly understood. Also, the existing matrix and tensor research is math-heavy and far away from machine learning (ML) and NLP research concepts. These two issues result in the recent progress on matrices and tensors for model parametrization being more like a loose collection of separate components from matrix/tensor and NLP studies, rather than a well-structured unified approach, further hindering algorithm design. To this end, we propose a unified taxonomy, which bridges the matrix/tensor compression approaches and model compression concepts in ML and NLP research. Namely, we adopt an elementary concept in linear algebra, that of a subspace, which is also the core concept in geometric algebra, to reformulate the matrix/tensor and ML/NLP concepts (e.g. attention mechanism) under one umbrella. In this way, based on our subspace formalization, typical matrix and tensor decomposition algorithms can be interpreted as geometric transformations. Finally, we revisit recent literature on matrix- or tensor-guided language model compression, rephrase and compare their core ideas, and then point out the current research gap and potential solutions.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 20 of its 31 references that are in the map. First posted on arXiv 3 Oct 2024.
Builds on
20 of 31 references.
References · 20
- 2017Attention Is All You Needcited by 59,932
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
- 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
- 2016On the Expressive Power of Deep Neural Networkscited by 302
- 2014The Loss Surfaces of Multilayer Networkscited by 278
- 2023Language Modeling Is Compressioncited by 225
- 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 137
- 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
- 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 100
- 2019A Tensorized Transformer for Language Modelingcited by 85
- 2019Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizationscited by 73
8 more
- 2020Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Mapscited by 36
- 2018GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinkingcited by 32
- 2021Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operatorscited by 28
- 2019Tensorized Embedding Layers for Efficient Model Compressioncited by 27
- 2021Kronecker Decomposition for GPT Compressioncited by 27
- 2024Compute Better Spent: Replacing Dense Layers with Structured Matricescited by 24
- 2022Exploring Extreme Parameter Compression for Pre-trained Language Modelscited by 15
- 2023Efficient GPT Model Pre-training using Tensor Train Matrix Representationcited by 8
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.