mlsanityData to 7 Oct 2026

Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators

arXiv 2106.02205 ↗First posted 4 Jun 2021cs.LG cs.AI quant-phcited by 28

Not on the map. Why?

Abstract (arXiv)

This paper presents a novel pre-trained language models (PLM) compression approach based on the matrix product operator (short as MPO) from quantum many-body physics. It can decompose an original matrix into central tensors (containing the core information) and auxiliary tensors (with only a small proportion of parameters). With the decomposed MPO structure, we propose a novel fine-tuning strategy by only updating the parameters from the auxiliary tensors, and design an optimization algorithm for MPO-based approximation over stacked network architectures. Our approach can be applied to the original or the compressed PLMs in a general way, which derives a lighter network and significantly reduces the parameters to be fine-tuned. Extensive experiments have demonstrated the effectiveness of the proposed approach in model compression, especially the reduction in finetuning parameters (91% reduction on average).

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 4 Jun 2021.

Builds on

33 of 46 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  11. 2017Learning multiple visual domains with residual adapterscited by 553
  12. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
21 more
  1. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  2. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  3. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  4. 2015Tensorizing Neural Networkscited by 246
  5. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  6. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  7. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  8. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 95
  9. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  10. 2019A Tensorized Transformer for Language Modelingcited by 85
  11. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  12. 2016Ultimate tensorization: compressing convolutional and FC layers alikecited by 55
  13. 2020SqueezeBERT: What can computer vision teach NLP about efficient neural networks?cited by 50
  14. 2020How fine can fine-tuning be? Learning efficient language modelscited by 41
  15. 2017Long-term Forecasting using Higher Order Tensor RNNscited by 34
  16. 2017Compressing Recurrent Neural Network with Tensor Traincited by 28
  17. 2019Tensorized Embedding Layers for Efficient Model Compressioncited by 27
  18. 2019Compressing deep neural networks by matrix product operatorscited by 27
  19. 2020schuBERT: Optimizing Elements of BERTcited by 17
  20. 2020A Model Compression Method with Matrix Product Operators for Speech Enhancementcited by 9
  21. 2020Grounded Compositional Outputs for Adaptive Language Modelingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 5; 2024: 9; 2025: 5; 2026: 5.

28 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  3. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43
  4. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  5. 2022Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Modelscited by 31
  6. 2023Tensor Networks Meet Neural Networks: A Survey and Future Perspectivescited by 29
  7. 2025LinkAlign: Scalable Schema Linking for Real-World Large-Scale Multi-Database Text-to-SQLcited by 26
  8. 2024QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptationcited by 23
  9. 2025Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrationscited by 16
  10. 2022Exploring Extreme Parameter Compression for Pre-trained Language Modelscited by 15
  11. 2024Propulsion: Steering LLM with Tiny Fine-Tuningcited by 14
  12. 2024Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Modelscited by 13
16 more
  1. 2024LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Modelscited by 12
  2. 2023On the Impact of Knowledge Distillation for Model Interpretabilitycited by 8
  3. 2024Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compressioncited by 7
  4. 2022Empowering parameter-efficient transfer learning by recognizing the kernel structure in self-attentioncited by 6
  5. 2025LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossingcited by 6
  6. 2024Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillationcited by 5
  7. 2024Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Modelscited by 3
  8. 2026HyperFM: An Efficient Hyperspectral Foundation Model with Spectral Groupingcited by 3
  9. 2022An Efficient Split Fine-tuning Framework for Edge and Cloud Collaborative Learningcited by 0
  10. 2023Scaling Pre-trained Language Models to Deeper via Parameter-efficient Architecturecited by 0
  11. 2024Application of Tensorized Neural Networks for Cloud Classificationcited by 0
  12. 2025CrossEarth-Gate: Fisher-Guided Adaptive Tuning Engine for Efficient Adaptation of Cross-Domain Remote Sensing Semantic Segmentationcited by 0
  13. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptationcited by 0
  14. 2026Fast Tensorization of Neural Networks via Slice-wise Feature Distillationcited by 0
  15. 2026A Hamiltonian-Inspired Local-Operator Ansatz for Slimming Large Language Modelscited by 0
  16. 2026Pre-registered tests of solid-state-physics-inspired LLM compression: a cluster-level negative result at small-language-model scalecited by 0